[SOLVED] What is the best (most performant) way to have encryption of VMs?

[gctwnl]

I've just started to experiment with proxmox.

I like my hardware to have encrypted data when they are turned off (data protected when hardware is stolen). I have an empty proxmox set up to which I'm going to add VMs (Ubuntu as the first one). What is the most secure and/or most performant way?

 

[Dunuin]

I guess most secure would be full system encryption but this is a bit annoying and complicated to setup with PVE. You could either install a Debian 11 with LUKS encrypted LVM/LVM-Thin (or even ontop of mdadm raid) and then install the proxmox-ve package ontop of it to convert the Debian into a PVE. Then setup initramfs-dropbear to unlock the root disk with SSH before PVE even is able to boot.
Or you install PVE from the PVE ISO as ZFS and later replace the unencrypted ZFS pool with an encrypted one (but this won't work with grub as bootloader).
Also don't forget to encrypt your swap partition and all the other storage you might want to add like additional VM/LXC storages or backup storage.
PVE won't support any encryption out of the box, so you will have to setup everything on your own using CLI.
Took me like 1 or 2 weeks to set that all up with full system encryption.

 

[leesteken]

Would enterprise Self Encrypting Drives be the most performant? They do the encryption themselves without bothering the CPU and would therefore scale.

EDIT: On a single drive with an i5 and only 8GB, I still think hardware encryption with ext4 would be faster than ZFS, but I guess you already bought the SSD.

 

[gctwnl]

Thank you, @Dunuin. I do prefer full encryption of the underlying filesystem (which means I have to reinstall PVE with ZFS). What is the consequence of "this won't work with grub as bootloader"?

(I'm new to both practical bare metal virtualisation and to Linux specifics (I have non-Linux Unix-like experience))

 

[gctwnl]

Would enterprise Self Encrypting Drives be the most performant? They do the encryption themselves without bothering the CPU and would therefore scale.

Probably. But for now I'm just setting up a NUC i5 with 256GB/8GB to be used for a few VMs, assuming local data on that SSD and continuity provided by backups (mechanism to be decided).

Zo I guess I have to first reinstall PVE and choose ZFS.

 

[Dunuin]

What is the consequence of "this won't work with grub as bootloader"?

You need to boot from a ESP. So no old hardware using BIOS or UEFI with ECM enabled could be used.
And ZFS needs alot o RAM. Don't wonder if you only got 2GB of your 8GB usable for guests. PVE needs 2GB and ZFS will use by default up to half of your RAM for caching.

EDIT: On a single drive with an i5 and only 8GB, I still think hardware encryption with ext4 would be faster than ZFS, but I guess you already bought the SSD.

ZFS in general wastes a lot of performance because of all the overhead and sync writes. But at least with SATA SSDs and a CPU that supports the AES-NI instruction set I couldn't see any big performance drop between ZFS without encryption and ZFS with enabled AES encryption. But encryption doubled the write amplification, so SSDs will die double as fast.

Here are some guides:
https://herold.space/proxmox-zfs-full-disk-encryption-with-ssh-remote-unlock/
https://gist.github.com/yvesh/ae77a68414484c8c79da03c4a4f6fd55
https://forum.proxmox.com/threads/encrypting-proxmox-ve-best-methods.88191/#post-387731

 

[gctwnl]

Would enterprise Self Encrypting Drives be the most performant? They do the encryption themselves without bothering the CPU and would therefore scale.

EDIT: On a single drive with an i5 and only 8GB, I still think hardware encryption with ext4 would be faster than ZFS, but I guess you already bought the SSD.

Actually, I have no problem with buying another SSD. So I think I’m going to follow this much less complicated solution. Anything to watch out for in combination with proxmox?

 

[leesteken]

Actually, I have no problem with buying another SSD. So I think I’m going to follow this much less complicated solution. Anything to watch out for in combination with proxmox?

Find an enterprise one that has a Power Loss Protection (or maybe that's a vulnerability?) for Proxmox. Make sure it works with your motherboard (for password input) and that is actually safe: implementation confirmed by external trustworthy party and not just password protected.
Did you read the potential problems in the Wiki article? I have no experience with such drives and know nothing about them, so don't attach much value to my opinions.

 

[LnxBil]

But encryption doubled the write amplification, so SSDs will die double as fast.

Oh, why that?

 

[Dunuin]

Oh, why that?

No, idea. But doing the same fio benchmarks of the same pool, with only difference non encrypted vs encrypted datasets/zvols, and looking at the "NAND writes" and "host writes" SMART attributes before and after the benchmark, enableing native encryption (no matter what encryption or hash algorithm) I have always seen the double amount of written data.

 

[gctwnl]

Find an enterprise one that has a Power Loss Protection (or maybe that's a vulnerability?) for Proxmox. Make sure it works with your motherboard (for password input) and that is actually safe: implementation confirmed by external trustworthy party and not just password protected.
Did you read the potential problems in the Wiki article? I have no experience with such drives and know nothing about them, so don't attach much value to my opinions.

And here I was thinking that self-encrypting drives would make my life easier...

Yes, I read the article.

No, idea. But doing the same fio benchmarks of the same pool, with only difference non encrypted vs encrypted datasets/zvols, and looking at the "NAND writes" and "host writes" SMART attributes before and after the benchmark, enableing native encryption (no matter what encryption or hash algorithm) I have always seen the double amount of written data.

Maybe the original data needs to be stored first before the encryption logic can calculate the new encrypted data and then the original data gets overwritten by the calculated encrypted data.

 

[Dunuin]

Maybe the original data needs to be stored first before the encryption logic can calculate the new encrypted data and then the original data gets overwritten by the calculated encrypted data.

Yes, but would be stupid to do that. You don't want to store unencrypted data on the disks, even if it is just temporarily. Especially when using SSDs there is no way to overwrite or delete data once it is written without doing a complete "secure erase" of the SSD (and that also only if the manufacturer didn`t screwed the algorithm up).

 

[gctwnl]

I was able to get the BIOS to handle this for me (self-encrypting NVMe M.2 SSD Samsung 980 PRO on Intel NUC 10th generation i5)

 

[gctwnl]

My conclusion: a self-encrypting drive is the most performant solution. These drives encrypt by definition everything with a key that lives on the drive. This key can be encrypted itself (protected by a password). This is what bitlocker does, for instance. I was able to do this with a BIOS from a 10th generation Core i5 Intel NUC and a Samsung 980 PRO NVMe M.2