Hey everyone. I made a script that makes it easy to do native ZFS root encryption for Proxmox. It optionally uses tailscale (or sshd) to allow remove unlocks. Check it out, let me know what you think: https://github.com/midzelis/zquickinit
Full system encryption with network unlock?
- Thread starter Joa
- Start date
Looks very interesting. For your "Security Considerations" you could maybe mention that when encrypting an unencrypted rpool the unencrypted data might still be recoverable. Especially when using SSDs there is no way to actually destroy the unencrypted data in the spare area without initializing a "secure erase" command which then would factory reset the whole SSD including the encrypted rpool. So not that useful unless you got a fresh unencrypted rpool that never got any sensitive data on it.
I would always propery wipe the disks and then install an encrypted PVE from scratch.
Features I would really like to see are:
a.) snapshot and rollback of the rpool
b.) host backups and restores to/from PBS
I would always propery wipe the disks and then install an encrypted PVE from scratch.
Features I would really like to see are:
a.) snapshot and rollback of the rpool
b.) host backups and restores to/from PBS
Last edited:
Thanks for your feedback! I'll definately make a remark in the doc. If you happen to be booting the system from a USB drive/PXE - you could theoretically do the SSD secure wipe command. In research, it seems that this is relatively complex to pull off, and varies based on SATA vs SAS vs NVME - so many best to just leave it in the notes, and maybe in the prompts. I could at least include the hdparm/nveme-format/blkdiscard binaries which could help.
ZFSBootmenu does support snapshots/rollbacks of the pools in the UI. My script will actually use ZFS checkpoints when encrypting an existing pool - assuming you'd be ok with the risk of recovering encrypted data.
During the install process, zfs snaphots are used before the process starts, after stage1 (debootstrap), and after stage2 (chroot package install).
I don't use PBS, but I'd be happy to accept PRs! ;-)
Yes but being able to boot into that ZFSBootmenu to do a manual snapshot or do a rollback would be a very useful recovery feature. Right know there is no official way to backup the PVE host before doing a upgrade nor is it possible to downgrade a PVE. So being able to create a snapshot before doing a major upgrade and a boot menu to easily rollback the rpool/ROOT/pve-1 to a state before that upgrade would be a really nice feature.
You should try it. So much better than VZDump. Faster because of incremental backups and dirty-bitmapping and sooo much more space efficient because of the deduplication. And the ransomware protection, integrity checks, encryption and option to sync multiple PBS are great too.
i'm sad that full system encryption is being discussed / asked for again and again when we don't even have complete backup encryption. which is low hanging fruit and should not be hard to implement.
why does nobody ask for that ?
https://bugzilla.proxmox.com/show_bug.cgi?id=4602
https://bugzilla.proxmox.com/show_bug.cgi?id=1560
why does nobody ask for that ?
https://bugzilla.proxmox.com/show_bug.cgi?id=4602
https://bugzilla.proxmox.com/show_bug.cgi?id=1560
Have you checked some other bugs lifecycle on Bugzilla? There's like 10 year old bugs for which there's a patch supplied, but not applied by PVE team. When you ask for that in a forum, there's silence. I could only infer that pushing new features is more important for PR than fixing bugs. Happened to many good projects before, unfortunately it never ended up well.
Re LUKS there was a nice tutorial made not too long ago:
https://forum.proxmox.com/threads/adding-full-disk-encryption-to-proxmox.137051/
There are also problems upstream where the PVE developers can't do much. The guy who primarily worked on ZFS encryption for example isn't actively working on it anymore and there is a missing features with ZFS native encryption that won't allow migrations in PVE when an encrypted ZFS pool is involved. So if you want stuff like migration working you will have to skip ZFS native encryption and do ZFS on top of LUKS.
And this then again makes stuff more complex. Would for example be more complex in case you ever need to replace a failed LUKS encrypted ZFS partition.
And this then again makes stuff more complex. Would for example be more complex in case you ever need to replace a failed LUKS encrypted ZFS partition.
Last edited:
Can you post more specific link? I can't find anything with a quick search and it does not sound related. My issue with how PVE uses ZFS was always that even backups do not take advantage of anything ZFS native hence completely defeating the purpose. The mantra reply was that it has to be "universal" for other filesystems too...
See the bug ticket "FS->ZFS storage_migrate does not work if zfs feature@encryption=enabled" from 2019 : https://bugzilla.proxmox.com/show_bug.cgi?id=2350
Last edited:
Whats the benefit of encrypting a probably 24/7 running system? I did it as well until somebody explained me how useless it is with something running 24/7 because everything is in memory anyway...
As well I gave up on ZFS this year because the ressources in comparsion to gain is not a ratio.
As well I gave up on ZFS this year because the ressources in comparsion to gain is not a ratio.
Three situations where it helps:
1.) in case someone gets physical access to your server. Not a problem in the datacenter but for a homeserver it isn't that unreasonable that some crackhead breaks into your home and steals something that looks expensive. Something like a tiny NUC running PVE would be such a thing. As soon as he steals it he has to unplug it from power and then all keys in RAM will be lost and all disks encrypted again. The person why buys the stolen good then at least can't access your private data.
2.) you got a disk that starts failing. You can't wipe that disk anymore because of the write errors but it is still readable. With an unencrypted disk you can RMA it but then people can access your data. If you need to care about data protection you would need to destroy that disk instead of sending it in for a free repair/replace. With an encrypted disk an RMA wouldn't be an problem as all the data is unusable once the disk is remove from the server.
3.) you don't have to unlock the disks 24/7. You can encrypt single datasets/zvols and only unlock them when needed. This at least helps a bit in case your server gets compromised while some of the data is still locked.
Last edited:
This make's literally no sense. You are afraid that somebody is breaking into your home and steal specifically your nuc because it is looking expensive? In the next sentence you have no problem when your server is running in a datacenter with hardware you probably don't even own and got your hosting provider which can access your system any moment without your knowledge and copying your memory and temper around?
The RMA thing is the only part I am with you but if you want to live up your privacy you have to give up this RMA stuff and descruct it. Then we talk about serious privacy.
This is a good example of how to do it. Separate and classify your data and utilize specific technology to cope with it.
Last edited:
Alright, I read through it, thanks for the link. Unfortunately I do not share the opinion this is an upstream issue by now in this case either - I am sure there might be some, but in my opinion this is a fail on PVE side. The reasoning given at https://bugzilla.proxmox.com/show_bug.cgi?id=2350#c6 is very weak. For anyone interested in why this "issue" exists in ZFS, the current link is: https://github.com/openzfs/zfs/issues/10507#issuecomment-651162104
I think the point is you get hardware stolen at times, when stolen for the value of the hardware, it's stolen in offline state. It's a legit reason.
If you do think encryption does not work, then of course you would never RMA anything and will not use encryption. Maybe it's NSA after you, who knows. NB Destructing something is not so trivial either.
This!
Secure Erase is nice and fancy, but only works on somewhat still working disks.
Thats why Distributors offer services like KYD (Keep Your Drive) to customers caring about their data (or being legallcy forced to care
)Used LUKS + ZFS years ago, with a thumbdrive connected to the machine that auto unlocks everything on boot.
Only to be able to rapidly buy, test and sell whatever disks, ignoring what was running on them. Homelab situation though.
Currently I do not encrypt my PVE setups but I also make sure the boot disks support Secure Erase, when the disk cannot successfully secure erase, I just keep it for good.
I am not afraid of anyone breaking into my home, but I really fear selling disks to whoever, thats why I hoard them.
There is always a higher level of paranoia available, whatever you need for your personal peace of mind or to comply with company policies.
You realise the Secure Erase is nothing else (on modern NVMe SSDs) than changing the keys for the locking range 0, i.e. there are always some keys in place on the self-encrypting drives, even if you do not use any setup in your firmware with TPM, etc. If you used it at all times, a drive failing would not matter, it would be unreadable outside of that system and can safely go for RMA.
It's about who got physical access. A trusted admin of a big hosting company that needs to pass a dozen of checkpoints to get to the server that is protected like a vault or some random hostile dude that simply breaks my window to get access while nobody is at home.