Install Proxmox VE / ZFS on a LUKS encrypted volume

[onlime]

AFAIK there is no easy solution to install Proxmox VE on ZFS on top of a LUKS encrypted volume. For us, both ZFS and encryption are strong requirements and we would prefer to use the Proxmox VE installer to get this up and running as fast as possible. Installing Proxmox VE on top of a plain Debian would also be fine, though (as described on https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie).

As ZFS on Linux does not seem to support native encryption in the near future, I would like to install the ZFS pools on top of a LUKS volume.

Can anybody please give me a hint how this could be done in a simple way?
Are there any plans to support hard disk encryption in the Proxmox VE installer any time soon?

 

[fabian]

AFAIK there is no easy solution to install Proxmox VE on ZFS on top of a LUKS encrypted volume. For us, both ZFS and encryption are strong requirements and we would prefer to use the Proxmox VE installer to get this up and running as fast as possible. Installing Proxmox VE on top of a plain Debian would also be fine, though (as described on https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie).

As ZFS on Linux does not seem to support native encryption in the near future, I would like to install the ZFS pools on top of a LUKS volume.

Can anybody please give me a hint how this could be done in a simple way?
Are there any plans to support hard disk encryption in the Proxmox VE installer any time soon?

I think your only choice at this point is

• boot some Debian-based live-environment with ZFS support (e.g., Ubuntu >= 16.04, or Debian Stretch)
• install ZFS user space (and compile and modprobe kernel module in case of Debian)
• install debootstrap and cryptsetup
• format your disks (think about how you want to boot - EFI or legacy? external boot device or encrypted /boot or separate unencrypted /boot on ZFS disks?)
• encrypt the partitions you want encrypted
• map them
• follow the ZFS on root tutorial from ZFS on Linux
  • you can probably reduce the number of datasets suggested - PVE does root, swap and data by default
    
  • debootstrap Debian Jessie (for PVE 4.x) or Debian Stretch (for PVE 5.x
  • instead of installing Debian's ZFS packages, crossgrade to PVE
  • make sure you have installed cryptsetup in your chroot, and update the initramfs
• reboot and be prepared to fix any issues you encounter

 

[morph027]

Would be nice, but unfortunately no one is doing this right now, due to the nature of complexity. We do use very small SSDs for Proxmox OS only and then store all other data on separate Luks+ZFS SSDs. So this is easy to maintain and allows us to use default Installer. And VMs and Data are secured. Passphrase is being asked on boot.

 

[DayBlur]

I played around with this a bit about a year ago and (from what I can remember) had some limited success. Specifically, I wanted to install onto an encrypted ZFS mirror (SSDs). If you are trying to install to a single encrypted ZFS drive, the following is not relevant.

My general approach was to use the normal proxmox installer to install (unencrypted) onto the ZFS mirror. Once installed, I booted into proxmox normally and proceeded to remove the drives from the zpool one at a time, re-partition and configure them for LUKS encryption, then add the opened encrypted volumes back to the zpool. After the quick rebuild of the first, I could do the same with the other drive. In this way, I was able to do the encryption migration online and was then running from an encrypted ZFS pool.

The biggest problem I ran into (and couldn't solve) was to get a first stage grub loader which could boot from ZFS and LUKS. It seemed like it should be possible, and at one point I thought I had it. All of the required modules (zfs, crypt, etc) were being compiled into the first stage (I had to remove some other things to get it all to fit) and testing grub from within linux, it seemed able to access and unlock the file system correctly with the grub loader I had built. Unfortunately, I never got it working at boot.

However, I was able to work around this by creating a small unencrypted /boot partition (I believe a standard ext2 RAID mirror). Set up the boot partition, move the /boot files over from the rpool and do whatever else was necessary (update mount points, re-install grub, etc).

Sorry for the lack of details, but this might at least give you an avenue of exploration. In the next month or so I plan to try this again with Proxmox VE v5.0 and should be sufficiently refreshed on the process to answer questions or possibly put together some notes.

Best,
James

 

[fabian]

a fully encrypted ZFS rpool (including /boot) works on Debian >= Stretch with just a little bit of tinkering (from memory, so I might have missed something): you need to add an entry to your crypttab for each encrypted device, and set GRUB_ENABLE_CRYPTODISK in /etc/default/grub. if you don't want to have to enter your passphrase twice for each encrypted device (once for Grub, once for the initramfs), you can slightly weaken security and add keyfiles for each of them to your (encrypted) initrd, and add the "initramfs" option to the crypttab entries. IIRC update-initramfs will print some warning about not being able to determine the rootfs, but it works nevertheless.

this setup quickly becomes a PITA for more than two disks (not related to ZFS specifically, it's just the way GRUB handles LUKS): imagine having to enter a passphrase for 16 disks twice each upon reboot, and a single typo means starting all over! in that case you'd probably go with a small (1 or 2 disk) rpool, and a big (encrypted) data pool that can be mounted (semi-)automatically when the system is booted. it is nice for something like a laptop though

 

[morph027]

I'm using a derived key for all dependent disks, therefore i have to enter only once: http://gw.tnode.com/debian/issues-and-workarounds-for-debian-8/

 

[fabian]

I'm using a derived key for all dependent disks, therefore i have to enter only once: http://gw.tnode.com/debian/issues-and-workarounds-for-debian-8/

but that only works with unencrypted /boot, or has Grub gained support for derived keys?

 

[morph027]

Jip, but i can live with that

 

[Ovidiu]

AFAIK there is no easy solution to install Proxmox VE on ZFS on top of a LUKS encrypted volume. For us, both ZFS and encryption are strong requirements and we would prefer to use the Proxmox VE installer to get this up and running as fast as possible. Installing Proxmox VE on top of a plain Debian would also be fine, though (as described on https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie).

As ZFS on Linux does not seem to support native encryption in the near future, I would like to install the ZFS pools on top of a LUKS volume.

Can anybody please give me a hint how this could be done in a simple way?
Are there any plans to support hard disk encryption in the Proxmox VE installer any time soon?

I was also looking for the same but this seems to be a total hassle to achieve or at least get close to so my thoughts are:

• I could give up on using ZFS as a raid solution, instead use hardware raid which makes it easy to install a luks encrypted proxmox and use i.e. thin-lvm which offers similar features to what ZFS would
• Again, give up using ZFS as a raid, instead use hardware raid which makes it easy to install a luks encrypted proxmox and then use ZFS simply as a file system and not using any "raid features" it offers to benefit from a few of its other features like snapshot syncing.

Do my 2 alternatives sound sound or did I misunderstand things?

 

[Nemesiz]

I cannot enter password for LUKS at boot time so I did this:

1. Disk partitions (for 3x SDD)
* bios boot
* ZFS mirror /boot
* LUKS ZFS RAIDZ /
* LUKS ZFS RAIDZ /data

2. Prepared GRUB with password protection for 'e'

3. After OS was prepared I set up Mandos ( https://wiki.recompile.se/wiki/Mandos ) for automatically password input.

Af course I know they way it may be hacked but its the way i can use only.

If where are direct access to enter password for LUKS I go like this:

1. Disk partitions (for 3x SDD)
* bios boot
* LUKS ZFS RAIDZ ( / /boot ... )

....

 

[Dennis Busch]

We came over that topic for one our customers from our it security department a time ago. We got quite deep in this. Our strong recommendation is to use SED - self encrypting devices. In fact, a lot of server HDD are SED and only depend on a supporting BIOS. LUKS is great for older setups, but if you buy new servers, look out for SED-capable Mainboards and/or disk controllers.

 

[Ovidiu]

We came over that topic for one our customers from our it security department a time ago. We got quite deep in this. Our strong recommendation is to use SED - self encrypting devices. In fact, a lot of server HDD are SED and only depend on a supporting BIOS. LUKS is great for older setups, but if you buy new servers, look out for SED-capable Mainboards and/or disk controllers.

I have a SED compatible server but the hosting company can't guarantee SED capable disks. They said its highly likely I will get some when ordering but not guaranteed.

Btw. is this correct:

only depend on a supporting BIOS

- so an SED-capable mainboard/bios + HD will do the encryption no matter what OS is running on the server?

 

[zenny]

I cannot enter password for LUKS at boot time so I did this:

1. Disk partitions (for 3x SDD)
* bios boot
* ZFS mirror /boot
* LUKS ZFS RAIDZ /
* LUKS ZFS RAIDZ /data

2. Prepared GRUB with password protection for 'e'

3. After OS was prepared I set up Mandos ( https://wiki.recompile.se/wiki/Mandos ) for automatically password input.

Af course I know they way it may be hacked but its the way i can use only.

If where are direct access to enter password for LUKS I go like this:

1. Disk partitions (for 3x SDD)
* bios boot
* LUKS ZFS RAIDZ ( / /boot ... )

....

@Nemesiz, Thanks for sharing. Just wanted to know whether you did this in PM4 or PM5? And can you throw some light on 2 above? What is 'e'?

It would be nice if you can share a small note on exact commands that you executed to accomplish this setup.

Cheers,
/z