Clarification on ZFS Encryption at Rest – Proxmox VE 9.2.3

[Rajan_Kumar.Jaiswal@sita.]

Hi Team,

We are currently evaluating Proxmox VE 9.2.3 for an Airport Edge use case and have a requirement for storage encryption at rest.
We understand that ZFS provides native encryption functionality, which is currently documented as an experimental phase. Before proceeding further with our evaluation, we would like to clarify the following:

1. Is native ZFS encryption at rest fully supported in Proxmox VE 9.2.3 for production environments?
2. Since ZFS encryption is documented as an experimental feature:
- What is the current support status of this functionality?
- Is there a planned timeline for making ZFS encryption fully production-ready?
3. If ZFS encryption is used in an HA environment:
- How are the encryption keys managed?
- How is the encryption key made available when a VM is migrated or failed over to another Proxmox node?
- Are there any limitations or recommended practices for key management in an HA configuration?
4. If native ZFS encryption is not currently recommended or supported for production HA deployments, what encryption-at-rest solution does Proxmox recommend for this use case?
5. Is there any beta/preview implementation or upcoming release that provides production-ready storage encryption at rest that we could evaluate?

Our primary requirement is encryption of data at rest, including VM and storage data, while maintaining supportability in an HA environment.

Could you please confirm the current supported approach and recommended architecture for this requirement?

 

[fba]

Hi,

do you have a specific reason for using the encryption feature of ZFS, instead of self-encrypting drives?

 

[jdancer]

I had a use case to mounting a non-boot/root encrypted ZFS pool. This was for a bare-metal Ubuntu machine. As you know, Proxmox is Debian with an Ubuntu kernel.

Called this systemd unit file "late-zfs-mount.service". I made sure it was the very last systemd unit file to run on the system, so that the other systemd unit files get processed. It didn't work if it was a beginning or middle systemd unit file. This systemd unit file loads the zfs key and mounts the pool.

The ZFS key is a random generated 32-bit key stored in /etc. The key itself has been copied off-system. The root filesystem itself is encrytped via LUKS with it's key stored on the CPU (AMD fTPM, Intel PTT) itself. I used the software package Clevis. You do have to regenerate initramfs/initrd to include the Clevis module.

This should work on Proxmox as long as it's not the root/boot ZFS filesystem.

You can avoid all of this by getting self-encrypting drives as was mentioned above.

 

[alexskysilk]

1. Is native ZFS encryption at rest fully supported in Proxmox VE 9.2.3 for production environments?

Yes, in the sense this is a feature that has a procedure to function in a linux system. If you specify By WHOM, you'd know who to ask specifically.

2. Since ZFS encryption is documented as an experimental feature:

If you mean per the Proxmox zfs on PVE pages, its "experimental" in the sense it is not within the PVE scope of support, at least in pve8. see above You would be dependent on non pve toolset for its administration.

3. If ZFS encryption is used in an HA environment:

ZFS is not a cluster filesystem. Consider it individualized per node. if you're looking for clustered storage with encryption, you'd need an external SAN or NAS that has the at-rest encryption option. If the intent is to use it for zfs replication, since the dataset is decrypted at boot it works normally.

4. If native ZFS encryption is not currently recommended or supported for production HA deployments, what encryption-at-rest solution does Proxmox recommend for this use case?

ZFS isnt recommended for production HA period. see above.

 

[tomservo]

You should consider some side effects of using encryption in different ways. Is it ok for all the VMs to share the same key? Is it ok for all the hosts to share the same key? Do you actually need the hosts encrypted too? Is it ok for the virtual disk to be readable by the host? Is it ok for the disk to be unlocked even when the VM is off? You haven't really explained the threat model you are trying to address with the encryption, so we can't answer those sorts of questions for you.

Depending on your answers to those questions there are lots of different ways to get disk encryption. For example, you might find that it is easier to do the encryption inside of the VMs. At my old job we made a Debian template with LUKS using a very simple password. When deploying a VM, you (well, ansible really, we didn't do these steps manually) would do these steps:
- Clone the VM, boot and unlock with that simple password
- "cryptsetup reencrypt" so this VM's encryption key is unique
- Grow the disk as needed
- Use clevis to bind the LUKS to a tang server. This way the password doesn't need to be typed on every boot, and the password isn't on the VM, and isn't sent over the network.
- Remove the initial simple password and replace it with the emergency password (the password we can use to unlock the disk if mounting manually, or if tang server is down)
- Test unlocking with both tang and the emergency password
- Set up whatever other software the VM needs

This aligned the process closely to the one used for hardware machines, and worked regardless of the storage on the host (zfs, ceph, lvm, whatever).