Allow migration and replication of disks on ZFS encrypted storage

[selbitschka]

Hi,

I came across the problem, that migration and replication does not work if the disk(s) are on a encrypted ZFS storage - which is not well documented by the way.

The problem arose since the ZFS export function in pve-storage use the -R option of zfs send and this option does not work with encrypted datasets unless you use -w as well. Since this is no option because we cannot transfer the data encrypted, since the target pool has a different encryption key etc., the question is why use -R in the first place?

Some discussion about the features of -R

1. Dataset properties are implicit included if using -R. But which dataset property of an ZVOL needs to be synced? Assuming my cluster members are identical configured, I see no need to copy properties.
2. All decent file systems are copied it using -R. Can a ZVOL of a vm-disk have decent file systems in proxmox? I don't think so.
3. Clones are also preserved it sending a dataset/snapshot with -R. Maybe this could be a pitfall.

After all this discussions I just tried it by patching /usr/share/perl5/PVE/Storage/ZFSPoolPlugin.pm


746,752c746
< my $cmd = ['zfs', 'send'];
< my $encrypted = $class->zfs_get_properties($scfg, 'encryption', "$scfg->{pool}/$dataset");
< if ($encrypted !~ m/^off$/) {
< push @$cmd, '-v';
< } else {
< push @$cmd, '-Rpv';
< }
---
> my $cmd = ['zfs', 'send', '-Rpv'];

This patch checks if the dataset is encrypted and omits -R and -p option during zfs send if so. This leads to an unencrypted stream of data, which is totally fine since our target pool will be encrypted as well.


I just tried this patch with hot and cold migration, replication, with snapshot etc. and everything works like a charm.


Question is: Do I miss something or is it possible to consider this patch be included?

I also opened an issue on github (https://github.com/proxmox/pve-storage/issues/10), but since this forum has a greater audience I repost it here.


regards
stefan

 

[Dunuin]

Also see here:
https://bugzilla.proxmox.com/show_bug.cgi?id=2350

This is really a missing feature. I like all my server to use encrypted ZFS and without migration or replication forming a PVE cluster is quite useless.

 

[selbitschka]

Hi,

after some testing its time to improve my former solution.

One of the problem if you replicate by sending encrypted snapshots unencrypted is the change of your encryption parameters. This topic is discussed in ZFSonLinux community since incremental snapshot could brick your data if the IV of you encryption change etc. Even if they fixed some of the issues there will still be problems if you have multiple sources where you send incremental snapshots from.

Best way so solve all this is to make every ZVOL its own encryption root without inheritance and replicate with -w option to send a raw stream which included all encryption parameters.

If we send an replication stream with -w the dataset of our ZVOL became its own encryption root and does not have its key loaded. Therefore you need to make further adjustments to the import part of ZFSPoolPlugin, to change the keylocation to its parent and load the key.

IMPORTANT: This solution will only work if both datasets on boths PVE nodes does share the same encryption passphrase and its stored in a file.

To get this up and running you need to patch /usr/share/perl5/PVE/Storage/ZFSPoolPlugin.pm as followed:

746,752c746
<     my $cmd = ['zfs', 'send'];
<     my $encrypted = $class->zfs_get_properties($scfg, 'encryption', "$scfg->{pool}/$dataset");
<     if ($encrypted !~ m/^off$/) {
<         push @$cmd, '-Rpvw';
<     } else {
<         push @$cmd, '-Rpv';
<     }
---
>     my $cmd = ['zfs', 'send', '-Rpv'];
808,819d801
<     ### set key location and load key
<     my $encrypted = $class->zfs_get_properties($scfg, 'encryption', $zfspath);
<     if ($encrypted !~ m/^off$/) {
<         my $keystatus = $class->zfs_get_properties($scfg, 'keystatus', $zfspath);
<         if ($keystatus eq "unavailable") {
<             my ($parent) = $zfspath =~ /(.*)\/.*$/;
<             my $keylocation = $class->zfs_get_properties($scfg, 'keylocation', $parent);
<             my $keyformat = $class->zfs_get_properties($scfg, 'keyformat', $parent);
<             eval { run_command(['zfs', 'set', "keylocation=$keylocation", $zfspath]) };
<             eval { run_command(['zfs', 'set', "keyformat=$keyformat", $zfspath]) };
<             eval { run_command(['zfs', 'load-key', $zfspath]) };
<         }
<     }

Furthermore you need a service which loads all keys on startup, since pve does not do this and otherwise you VM could not start automatically, i.e. "/sbin/zfs load-key -r tank/vmdata_encrypted".

This solution avoids changing the key with "zfs change-key" like suggested in the bug report https://bugzilla.proxmox.com/show_bug.cgi?id=2350. Since changing the encryption key will brick subsequent incremental sends etc. To avoid all this every ZVOL is its own encryption root. This also allows you to send incremental backups from host A and host B after a failover, which is not possible if you have different keys or replicate unencrypted.

Keep in mind that the encryption root is changed on replication and not for all existing ZVOL. If you want to do this in advanced, i.e. to setup you backup you need to:
zfs get name,keylocation,keyformat tank/vmdata_encrypted
zfs change-key -l -o keylocation=XXX -o keyformat=YYY tank/vmdata_encrypted/vm-XXX-disk-Y

From my point of view the encryption of each ZVOL would also be the best solution to integrate this into the main product. This allows you to choose different encryption primitives for different VMs and gets rid of all the replication, backup and other sync problems which came with inherited encryption.

Does anybody see any drawbacks in making every ZVOL its own encryption root? Beside the fact that the initial key load took some time if you have many ZVOLs.

regards
stefan