Host level LUKS encryption on entire Proxmox setup

[kartr]

I plan to deploy Proxmox VE 8 on a dedicated server meant to host multiple VMs with sensitive information and I have 2 main requirements

1. How to encrypt the entire server with LUKS so that I enter a password on boot manually and that is it and I don't have to deal with encryption on per-VM level anymore?
2. I want KVM VMs and I want to use thin provisioning so that if a VM is allocated 100GB but uses only 5GB, it's snapshot/backup would be only 5GB in size and not 100GB.

I plan to use 2 drives in RAID0 on the host machine for this setup.

Are both goals above achievable? Looking forward for an answer from the community.

 

[santiagobiali]

Hi, you can use ZFS and encrypt the dataset, this way the VM's disk is encrypted and Proxmox won't turn the VMs on until you unencrypt it.

1. Create the dataset:
zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase rpool/vms
2. Add the ZFS storage to proxmox's GUI and use it for your vms.
3. Unencrypt at boot:
echo 'password' | zfs load-key -r rpool/vms

This way Proxmox isn't encrypted, only the VM's disks.

 

[kartr]

Hi, you can use ZFS and encrypt the dataset, this way the VM's disk is encrypted and Proxmox won't turn the VMs on until you unencrypt it.

1. Create the dataset:
zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase rpool/vms
2. Add the ZFS storage to proxmox's GUI and use it for your vms.
3. Unencrypt at boot:
echo 'password' | zfs load-key -r rpool/vms

This way Proxmox isn't encrypted, only the VM's disks.

What about VM backups/snapshots here? And how does it affect VM disk expansion? My original question remains to be FULL HOST LUKS encryption but still want to know your reply here.

 

[santiagobiali]

What about VM backups/snapshots here? And how does it affect VM disk expansion?

After you unlock the zfs dataset, it will function as a normal dataset. So you will have every feature you get with it, including snapshot, backup and replication.
VM disk expansion is simple, you just expand it in the Proxmox web gui, then inside the VM.

 

[Dunuin]

You could install a Debian 12 using the installer of the Debian 12 ISO. That installer supports LUKS while the PVE installer does not. For the raid0 you could use mdadm + LVM that the Debian installer is supporting. You then could install the PVE packages on top of that encrypted Debian 12: https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
For snapshots you would need to manually create a thin pool via CLI on that LVM later.

Btw...I wouldn't use raid0 for anything other than temporary files. If you really want to use it make sure to have proper backups of your host configs + guests as well as proper documentation.

 

[esi_y]

You can also run LUKS under ZFS (or under BTRFS, etc.).

I am not sure if RAID0 was a typo, but that's your call (maybe you don't care for downtime and are happy with fully manual recovery from backups). If you were to do mdadm, be aware of: https://bugzilla.proxmox.com/show_bug.cgi?id=5235

 

[Dunuin]

You can also run LUKS under ZFS (or under BTRFS, etc.).

But that's a very advanced topic. Debian Installer isn't supporting ZFS and PVE installer insn't supporting LUKS. From what I've read so far full system encryption using ZFS on LUKS is full of pittfalls and has all to be done manually via CLI from a Live Linux after installing an unencrypted PVE. Using ZFS native encryption for full system encryption is pretty well documented in this forum and blogs. But I couldn't remember a single head-to-tail tutorial for LUKS.
If someone got this working, I would be interested, as ZFS native encryption is preventing me from clustering and I would need to redo the hosts with ZFS+LUKS.

 

[esi_y]

But that's a very advanced topic. Debian Installer isn't supporting ZFS and PVE installer insn't supporting LUKS.

I don't mean to be harsh (I know you have more knowledge on storage here than myself), but basically the PVE installer in my view is so rudimentary I do not even consider it an issue if it does not support something. Meanwhile the PVE install on Debian works well.

LUKS setup under anything (that e.g. Debian installer does not support even) is very much the same:
https://forum.proxmox.com/threads/proxmox-8-luks-encryption-question.137150/#post-610704

From what I've read so far full system encryption using ZFS on LUKS is full of pittfalls and has all to be done manually via CLI from a Live Linux after installing an unencrypted PVE.

I don't know if that's (adding ZFS to Debian) advanced, it's all the same once people run the setup - I think it's better they understand it by setting it up manually (if that's what they want, i.e. installer should not stop them).

But for full disclosure, I really do not like ZFS for the system drive, so that is one major issue out of the way. This does not stop anyone to use ZFS (or what they like) for the non-root pool.

Using ZFS native encryption for full system encryption is pretty well documented in this forum and blogs. But I couldn't remember a single head-to-tail tutorial for LUKS.

To make a tutorial is the least of a problem. One can install root on ZFS, the LUKS is just extra layer underneath, i.e. if you can add LUKS, you can get in the kernel module for ZFS as well. If you can't, you are better off not doing it as troubleshooting later is a nightmare. For me it's not worth it (the troubleshooting) for no real benefit.

Native ZFS encryption is not mature, besides it does not encrypt metadata. For me the feature was DOA.

If someone got this working, I would be interested, as ZFS native encryption is preventing me from clustering and I would need to redo the hosts with ZFS+LUKS.

What's the benefit of ZFS for root for you? In a cluster ... especially.

 

[esi_y]

If someone got this working, I would be interested.

Here you are basically, combine with the LUKS from thread above for the vdevs and you are all set:

https://www.dwarmstrong.org/debian-install-zfs/

 

[Dunuin]

What's the benefit of ZFS for root for you? In a cluster ... especially.

I like all my disks mirrored via software raid and with checksumming. The only other supported option would be btrfs and thats still in "feature preview" state, less mature and got other problems. The unsupported mdadm works but got problems too (like the bug you linked above).
Encryption on PVE already is a very hacky approach and I prefer to use the stuff that is most tested (so LVM-Thin or ZFS).

 

[esi_y]

I like all my disks mirrored via software raid and with checksumming. The only other supported option would be btrfs and thats still in "feature preview" state, less mature and got other problems. The unsupported mdadm works but got problems too (like the bug you linked above).
Encryption on PVE already is a very hacky approach and I prefer to use the stuff that is most tested (so LVM-Thin or ZFS).

Apparently PVE expect everyone to be running FIPS 140-2 SED drives to abstract from it.

I would just like to point out that if you e.g. use the mdadm only for the system drive, you don't really worry for those qemu related things in relation to it.

Also strictly speaking, yet another option would be ... LVM mirror. I think it will work just fine, but it is something I have not seen being used (unlike mdadm is battle tested). BTRFS is "preview" as you say, I think we all know it will be a "forever-preview." Just as ZFS will be non-free Debian due to its license (and so never support ZFS on root, even Ubuntu gave up on that). The biggest practical issue with root on ZFS will always be the bootloader out of the box support (the linked guide used yet a different approach compared to stock PVE install).

I just want to point out OpenZFS is immature too in many aspects, as in, there are bugs (as always with something in active development, as people like to start using new features early). (E.g. [1][2][3])

The sad part with LUKS is that it is really well tested, reliable and all it takes is to specify custom mountpoints at install. It would be a tick box for PVE installer, if they wanted.

EDIT:

Oh and one more thing - now that ZFS finally got reflinks [4], you can't obviously use them across encrypted datasets. So just consider ZFS native encryption is never really an alternative to FDE.

[1] https://github.com/openzfs/openzfs-docs/issues/494
[2] https://github.com/openzfs/zfs/issues/14533
[3] https://github.com/openzfs/zfs/issues/15351
[4] https://github.com/openzfs/zfs/issues/405