ZFS on-disk encryption & VM Templates

[diverseprox]

Down another rabbit hole I go. This time, on ZFS on-disk encryption. I was surprised this is not enabled by default, but I'm a noob, so bare with me.

You can run the follow to set encryption to on:
zfs create \
-o encryption=on \
-o keyformat=passphrase \
-o keylocation=prompt \
tank/backups-encrypted

I'm assuming there is an overhead cost associated with this, which is likely the reason it's not enabled by default?

----

I created a Windows template and when running `sysprep.exe`, it gave me pushback related to `BitLocker` and a `Microsoft.WidgetsPlatformRuntime`. No problem, I disabled BitLocker and decrypted the drive. I then removed the widget for all users. Once the decryption completed, I was able to run sysprep.exe and create the template. When creating a new VM based on that template, I now have all of the default apps I installed and settings which is great. I can create a new VM/user quickly.

Once the new VM is deployed and user is created, I need to re-enable BitLocker and encrypt the drive. This isn't so time consuming I can't do it. In fact, just running with qm agent makes it so I don't even have to run in PowerShell within the VM. One extra step that can be easily scripted. At some point, maybe I'll also use unattend.xml to quickly populate the VM user, etc.
----

But, this got me thinking. If ZFS can be encrypted at rest, why encrypt with BitLocker at all? These VMs live on Proxmox and ZFS, so as long as ZFS is encrypted, I shouldn't actually need to re-enable BitLocker, right? No extra step that takes extra time rebooting. Hell, if they're lightly used and not a heavy-write user, I could even get one of these stood up in a matter of seconds, using a linked clone. Fully loaded. I understand that there can be problems related to links breaking, etc. Depending on the use case, it's better just to go with the full clone.

So, the questions are:
1. Why is ZFS encryption not on by default?
2. If you enable ZFS encryption, do you agree that you don't actually need BitLocker re-enabled?
3. Do you encrypt your ZFS pools/datasets? Why or why not?
4. Do you tend to just use a linked clone VM, or go with a full clone?

---

Thanks in advance. I'm liking this community already

 

[cheiss]

There is actually already a section in our admin guide about that: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#zfs_encryption

1. Why is ZFS encryption not on by default?

As mentioned in our admin guide, ZFS encryption can cause problems when used in the context of virtualization. You cannot replicate encrypted datasets and snapshots are somewhat known to be problematic. And these are two features for which ZFS is often choosen as filesystem.

Also, booting from encrypted pools is a whole other can of worms.

2. If you enable ZFS encryption, do you agree that you don't actually need BitLocker re-enabled?

IMHO no definitive answers exist here, as it depends on the threat model. Just look at e.g. backups, if you do unencrypted backups - then you break the encrypted-at-rest paradigm again.

3. Do you encrypt your ZFS pools/datasets? Why or why not?

From my completely personal usage: I do encrypt some pools which just hold data, but would never put VMs on encrypted pools.

4. Do you tend to just use a linked clone VM, or go with a full clone?

As long as you don't plan on removing the template VM, there aren't really downsides to using a linked clone when on ZFS. Depends again on your specific situtation.

 

[FrankList80]

My 2 cent: Encryption of (VM-Block) Storage at Rest for VMs, which are 24/7 online is indeed often a regulatory requirement, but an OPS nightmare, with imo VERY questionable logic. How often is the storage layer attacked and when the OS? If i have access to a running VM its already too late. And in DR cases this is another nightmare to have the keys present. Better look at Application Level Encryption or something like this.

 

[diverseprox]

Okay, thank you for your responses @cheiss and @FrankList80. I'll take a further look at the guide you linked.

As mentioned in our admin guide, ZFS encryption can cause problems when used in the context of virtualization. You cannot replicate encrypted datasets and snapshots are somewhat known to be problematic. And these are two features for which ZFS is often choosen as filesystem.

Also, booting from encrypted pools is a whole other can of worms.

When I was reading more about this yesterday I was leaning towards it's better to possibly encrypt certain datasets if you want that additional layer of security, but encrypting everything is generally a bad idea.

IMHO no definitive answers exist here, as it depends on the threat model. Just look at e.g. backups, if you do unencrypted backups - then you break the encrypted-at-rest paradigm again.

I'll take a look into this a little more carefully, but I think I'll proceed with the plan of only encrypting certain datasets, and re-enable BitLocker on the clones and encrypt.

From my completely personal usage: I do encrypt some pools which just hold data, but would never put VMs on encrypted pools.

I'll follow this advice.

As long as you don't plan on removing the template VM, there aren't really downsides to using a linked clone when on ZFS. Depends again on your specific situtation.

Okay, I went ahead with the cloned instances yesterday as well. That's good to know re: leaving the template intact.

----

When it comes to Windows VMs and BitLocker, I'll just have the script encrypt the instances again after replication. I'm not running a service like AWS where you expect it to spin up in about 30 seconds -- the linked clones are near-instant and re-enabling BitLocker and encrypting should take just a few minutes (with a quick script). I'll probably add in the unattend.xml so it's automated for filling in the few parameters in the clone.