Need help installing Proxmox with automatic decryption and multiple drives

[needzfshlp]

I'm trying to install Proxmox on a server that is going to be running Home Assistant, a security camera NVR setup and other sensitive data, I need to have the drives be encrypted with automatic decryption of drives so the VMs can automatically resume after a power failure.

# My desired setup:

* 2 Sata SSDs boot drives in a ZFS mirror
* 1 NVME SSD for L2ARC and VM storage
* 3 HDDs in a RAIDz1 for backups and general large storage
* 1 (maybe more added later) HDD for Camera NVR VM.

I'd prefer every drive encrypted with native ZFS encryption automatically decrypted by either TPM 2.0 or manually by a passphrase if needed as a backup.

# Guide I found:

I found a general guide on how to do something similar but it honestly went over my head (I'm still learning) and didn't include much information about additional drives: Proxmox with Secure Boot and Native ZFS Encryption


If someone could adapt that post into a more noob friendly guide for the latest Proxmox version, with directions for decryption of multiple drives, that would be amazing and I'm sure it would make an *excellent* addition to the Proxmox wiki

# My 2nd preferred setup:

* 2 Sata SSDs boot drives in a ZFS mirror with LUKS encryption and automatic decryption with clevis.
* All other drives encrypted using ZFS native encryption with ZFS key (keys?) stored on LUKS boot drive partition.


With this arrangement, every drive could be encrypted at rest and decrypted on boot with native ZFS encryption on most drives but has the downsides of using LUKS on ZFS for the boot drives.


Is storing the ZFS keys in a LUKS partition insecure in some way? Would this result in undecryptable drives if something happened to ZFS keys on the boot drive or can they be also decrypted with a passphrase as a backup?


As it stands right now, I'm really stuck trying to figure this out so any help or well written guides are heavily appreciated. Thanks for reading!

 

[Dunuin]

I found a general guide on how to do something similar but it honestly went over my head (I'm still learning) and didn't include much information about additional drives: Proxmox with Secure Boot and Native ZFS Encryption

Additional drives aren't a problem when you work with keyfiles. You of cause then need a secure location (like an encrypted root partition) to store those keyfiles. See for example here: https://forum.proxmox.com/threads/f...s-using-proxmox-installer.127512/#post-557808

Would this result in undecryptable drives if something happened to ZFS keys on the boot drive or can they be also decrypted with a passphrase as a backup?

A keyfile is just a bunch of characters. Once the keyfile is lost, you won't be able to unlock your disks. So back them up using a password manager, print them out and put them in a safe or whatever.

 

[LnxBil]

I need to have the drives be encrypted with automatic decryption of drives so the VMs can automatically resume after a power failure.

For what use case are you encrypting the drives if you store the key on the drives itself? If someone stole your machine, there is no protection at all.

We use LUKS and ZFS native encryption in all our non-local servers (e.g. hosting and housing) and decrypt via nagios/icinga monitoring after detecting a reboot. A simple check run remotly that checks if mounted otherwise decrypt using the remotely provided key. With this setup, the machine requires remote decryption, but it is the only way I know to secure it properly.

 

[needzfshlp]

For what use case are you encrypting the drives if you store the key on the drives itself? If someone stole your machine, there is no protection at all.

So traditionally the keys are stored in the TPM on the CPU. I'm having a hard time setting up native ZFS encryption where the keys are stored on the TPM. The second scenario that I described places the native ZFS encryption keys for the additional non-boot drives on an encrypted drive using LUKS on the boot drive which can be unlocked via the TPM. That is something I can do, but running ZFS on LUKS has additional downsides and is not preferable to native ZFS encryption. If someone stole the machine, it is protected by the login prompt. A metaphorical comparison could be made to a laptop that has full disk encryption but is stolen while it is powered on. The person who stole the laptop wouldn't be able to access the data because the laptop still has a login prompt of some kind.

So imagine a scenario where the power goes out, even just for a small amount of time. The uninterruptible power supply connected to the server allows for clean unmounting of the filesystems and then shuts the server off. The server then comes back on when power is detected from the grid again. If an encrypted drive doesn’t have automatic decryption and requires a passphrase before boot, the services that I’m self hosting aren’t running. I’m wanting to run home assistant and a security camera NVR so that could mean that I’m stumbling around in the dark tripping over things to get to the server to type in a passphrase because the lights didn't power on, or there could be a robbery and I now have no evidence of who the culprit might be because the server didn't send the pictures of someone breaking in to my phone and external backup source.


Having the drives automatically decrypt in a safe manner helps ensure higher availability (without me spending a small fortune in additional hardware costs because you can usually throw money at a problem to fix it), while still offering data protection in the event of a robbery.


The alternative to automatically decrypting drives while ensuring my services work after power failure is to not encrypt the drives at all.

 

[needzfshlp]

Additional drives aren't a problem when you work with keyfiles. You of cause then need a secure location (like an encrypted root partition) to store those keyfiles. See for example here: https://forum.proxmox.com/threads/f...s-using-proxmox-installer.127512/#post-557808


A keyfile is just a bunch of characters. Once the keyfile is lost, you won't be able to unlock your disks. So back them up using a password manager, print them out and put them in a safe or whatever.

I know normally when setting up automatic decryption using keys stored on the TPM of a LUKS install without using native ZFS encryption, you can specify that the drive can be decrypted with multiple sources like both a keyfile stored on the TPM or a passphrase. The drive would get automatically decrypted using the keys stored on the TPM, but you could still decrypt the drive using a passphrase (stored in a secure location) as a backup if the drives had to be moved to a different computer, or if the TPM fails, or some other scenerio where you couldn't use the keys stored on the TPM. Does native ZFS encryption allow for using multiple key sources to unlock native ZFS encryption? I'm having a hard time setting up native ZFS encryption where the keys are stored on the TPM. The second scenario that I described places the native ZFS encryption keys for the additional non-boot drives on an encrypted drive using LUKS on the boot drive which can be unlocked via the TPM. That is something I can do, but running ZFS on LUKS has additional downsides and is not preferable to native ZFS encryption.

 

[alexskysilk]

Having the drives automatically decrypt in a safe manner helps ensure higher availability (without me spending a small fortune in additional hardware costs because you can usually throw money at a problem to fix it), while still offering data protection in the event of a robbery.

umm... how is it providing data protection in the event of a robbery? the system will decrypt at boot; all that is necessary to get at the data is physical possession.

 

[needzfshlp]

umm... how is it providing data protection in the event of a robbery? the system will decrypt at boot; all that is necessary to get at the data is physical possession.

My threat model is not state level actors or master hackers. In the event of a robbery, the most likely thing to happen is the server is sold by some crackhead on an online local marketplace, or just the OS wiped and replaced with Windows.

But lets assume someone stole the server with the intention of accessing the data in another spot...

Secure boot uses the TPM with my custom keys ensures integrity of the boot files so they can't be tampered with. My motherboard UEFI is password protected. If someone clears the password by resetting the UEFI, the custom TPM keys are cleared. After boot, the encryption keys enter RAM. My motherboard UEFI supports memory encryption making reading the encryption keys in RAM significantly more difficult to get at the keys. When the system is booted, it drops to a login shell. The attacker should only have access to a login shell that he doesn't know the password to. In summary, the attacker can't tamper with any of the boot process because of the TPM and Secure Boot, can't tamper with the UEFI because it is password protected, can't clear the UEFI to remove the password without clearing the TPM keys, can't extract the keys from RAM because of memory encryption, and can't do anything past a login shell they don't know the password to. This is very similar how Bitlocker on Windows works.


Please explain how to get at the data on the drive. Keep in mind that my threat model is NOT state level actors.

 

[alexskysilk]

fair point. makes sense for your usecase.

 

[Dunuin]

I’m wanting to run home assistant and a security camera NVR so that could mean that I’m stumbling around in the dark tripping over things to get to the server to type in a passphrase because the lights didn't power on, or there could be a robbery and I now have no evidence of who the culprit might be because the server didn't send the pictures of someone breaking in to my phone and external backup source.

Why you then don't use a UPS so the server would continue running in case of a power failure? This is how you usually do it. Especially when running security cameras which should continue recording in case someone cuts the electricity line.

 

[needzfshlp]

Why you then don't use a UPS so the server would continue running in case of a power failure? This is how you usually do it. Especially when running security cameras which should continue recording in case someone cuts the electricity line.

You quoted me in your post here, but then conveniently left out the beginning of the paragraph you quoted which stated:

"So imagine a scenario where the power goes out, even just for a small amount of time. The uninterruptible power supply connected to the server allows for clean unmounting of the filesystems and then shuts the server off. The server then comes back on when power is detected from the grid again."


I have an uninterruptible power supply, but its small and mainly for safe shutdowns. It's not currently in the budget for a larger one.

This is my first post in this forum, I've done the best that I can to try to be informed as I can about proxmox before posting here and I know I'm still learning. I came here hoping to get some friendly advice with my technical issue from a welcoming community. A place where I could be humble and admit that I don't know how to do this. I've instead had multiple people argue about the way I'm trying to set this up as if the very question I'm asking is wrong.

I would appreciate if we could instead return to the original focus of the discussion, that being installing proxmox with automatic disk decryption using a TPM with native zfs encryption.

 

[LnxBil]

The person who stole the laptop wouldn't be able to access the data because the laptop still has a login prompt of some kind.

Thank you for the long explanation, but this is the crucial part: This may be true for windows, due to a lack of knowledge, but in Linux, there is nothing restricting this as long as you can inject whatever you want into the boot process. The TPM is used to decrypt your disks and this can be used from any Linux. All current implementations I saw get the key from the TPM on boot and use them to decrypt LUKS. As an adversary, you just boot your own linux environment, load the initramfs, which is unencrypted and inspect it. You then find the crypttab and decode what was done, reverse engineer it and load the LUKS key - do exactly that what would have been done by the OS/initram and voila ... decrypted. This is fine for securing the disk, but if your whole machine is stolen, it does not offer any additional security. If you also want to secure against such an attack, please read this, which summarizes it pretty well.

 

[LnxBil]

...and now we also know how to sniff the key while the OS / Windows is booting:

https://www.youtube.com/watch?v=wTl4vEednkQ

In addition to the video and sadly not mentioned there, for those TPM chips exist premade clamps that attach to the chip itself in order to have a very easy method to sniff it. I have a bunch of them for inplace flashing firmware, e.g. SOP-16 clamp.

 

[Dunuin]

I also remembered this discussion when watching that video some days ago but wasn't able to find this thread.