[TUTORIAL] Native full-disk encryption with ZFS

[Bugbear]

Hello there!

I really like the idea of having the full root-filesystem encrypted using native ZFS encryption (only).
However, as I currently couldn't find any complete writeup on on this topic, here's what worked for me (with and without Secureboot on PVE 8.1):

1. Install with zfs (RAID0 for single-disk application)
2. Reboot into ISO > Advanced Options > Graphical, debug mode
3. Exit to bash with `exit` or Ctrl+D
4. Execute following commands:

# Encrypt root dataset
zpool import -f rpool                        # Force import the ZFS pool named 'rpool'
zfs snapshot -r rpool/ROOT@copy              # Create a recursive snapshot of 'rpool/ROOT'
zfs send -R rpool/ROOT@copy | zfs receive rpool/copyroot            # Duplicate the snapshot to 'rpool/copyroot'
zfs destroy -r rpool/ROOT                    # Destroy the original 'rpool/ROOT' to replace it with an encrypted version
zfs create -o encryption=on -o keyformat=passphrase rpool/ROOT      # Create a new 'rpool/ROOT' with encryption
zfs send -R rpool/copyroot/pve-1@copy | zfs receive -o encryption=on rpool/ROOT/pve-1    # Restore 'pve-1' from the copy
zfs destroy -r rpool/copyroot                # Clean up by removing the temporary copy
zpool export rpool                           # Export the pool to finalize changes

# Prepare for chroot & destroy rpool/data dataset
zpool import -f -R /mnt rpool                # Import the pool with an alternate root at /mnt
zfs load-key -a                              # Load the encryption keys for all encrypted datasets
zfs destroy -r rpool/data                    # Destroy original dataset as after mounting pve-1 in the next step rpool/data will appear `busy` (see post #4 below)
zfs mount rpool/ROOT/pve-1                   # Mount the 'pve-1' dataset
mount -o rbind /proc /mnt/proc               # Recursively bind the /proc directory to the chroot environment
mount -o rbind /sys /mnt/sys                 # Recursively bind the /sys directory
mount -o rbind /dev /mnt/dev                 # Recursively bind the /dev directory
chroot /mnt /bin/bash                        # Change root into the new environment

# Create encrypt rpool/data dataset
dd if=/dev/urandom bs=32 count=1 of=/.data.key         # Create a new encryption key
chmod 400 /.data.key                                   # Set appropriate permissions for key
chattr +i /.data.key                                   # Make key immutable
zfs create -o encryption=on -o keylocation=file:///.data.key -o keyformat=raw rpool/data     # Create a new dataset with encryption enabled
# Setup systemd service for automatic unlocking of rpool/data on boot
sudo cat > /etc/systemd/system/zfs-load-key.service <<'EOF'
[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a

[Install]
WantedBy=zfs-mount.service
EOF
systemctl enable zfs-load-key

## Optional! Only needed if stuck at boot with ZFS-encryption enabled (see post #3 below): Update boot configuration
echo "simplefb" >> /etc/initramfs-tools/modules       # Add 'simplefb' to initramfs modules
update-initramfs -k all -u                            # Update all initramfs images
proxmox-boot-tool refresh                             # Refresh Proxmox boot configuration to apply changes

# Cleanup and reboot
exit
umount /mnt/proc                              # Unmount /proc
umount /mnt/sys                               # Unmount /sys
umount /mnt/dev                               # Unmount /dev (if target is busy, check for nested mounts)
zfs unmount rpool/data                  # Unmount the ZFS dataset
zfs unmount rpool/ROOT/pve-1                  # Unmount the ZFS dataset
zpool export rpool                            # Export the ZFS pool
Ctrl + Alt + Del                              # Use key combination to reboot the system

----
Credits to:
@nschemel https://forum.proxmox.com/threads/encrypting-proxmox-ve-best-methods.88191/#post-387731
@bindi https://forum.proxmox.com/threads/unable-to-unlock-zfs-root-dataset-during-boot.138172/post-616393
@Stoiko Ivanov https://forum.proxmox.com/threads/e...option-system-cannot-boot.104377/#post-450093
@yvesh https://gist.github.com/yvesh/ae77a68414484c8c79da03c4a4f6fd55
@Tommy Tran on PrivSec https://privsec.dev/posts/linux/using-native-zfs-encryption-with-proxmox/

PS: Search engines aren't what they used to be :/ ...

 

[Dunuin]

You forgot to encrypt the "data" dataset. So VMs/LXCs stored on your disk won't be encrypted.

Why is the "simplefb" required? As far as I remember I never did that when encrypting my rpool. I only read about "simplefb" in context of GPU passthrough.

In case you want to be able to unlock it headless I can highly recommend using the dropbear-initramfs SSH server.

 

[Bugbear]

Hey @Dunuin , thank you very much for the feedback! Dropbear is truely neat (and easy to set up with PrivSecs guide from above), personally I'll just use an IP-KVM.
Good catch with rpool/data! Totally forgot that ... I just added it to the guide.

As for the simplefb-module I wondered too, however without this solution on boot I get stuck at this screen (like @jwalker in this thread) with SecureBoot disabled:

and stuck at this screen with SecureBoot enabled:

However, while being stuck at these screens in the console, I can still unlock the zfs dataset with dropbear and then the OS loads perfectly fine (on the console too!).
So it's "just" a graphical issue with this message hanging on boot, fixable with the simplefb-module in the initramfs.

Cheers!

 

[dynostatic]

I followed these steps exactly and managed to get ROOT encrypted. Unfortunately, I ran into an error when trying to encrypt rpool/data. I made it as far as "zfs destroy -r rpool/data". This resulted in the error "cannot destroy rpool/data: dataset is busy". Not sure where to go from here.

PVE 8.1 w/ secure boot, x2 nvme zfs raid 1

 

[Bugbear]

Hi @dynostatic, I'm sorry for this issue. Please try the "Encrypt rpool/data" section before the "Prepare for chroot" section and thus before importing the pool.

EDIT: I just updated the guide so this workaround won't be needed anymore. Should work flawlessly now.

 

[dynostatic]

@Bugbear That pointed me in the right direction. Thank you!

 

[andre78]

Thank you @Bugbear for the instructions, I was able to set up a Proxmox host at at my hosting provider.

However, when trying to create a LXC, I am getting this error:

TASK ERROR: unable to create CT 100 - zfs error: cannot create 'rpool/data/subvol-100-disk-0': encryption root's key is not loaded or provided

Do you have a suggestion how to get around this?

Thank you!

 

[Bugbear]

Hi @andre78, you need to catch up on the # Setup systemd service for automatic unlocking of rpool/data on boot section for automatic unlocking of rpool/data on boot. Otherwise you'd need to execute `zfs load-key -a` on every reboot.

 

[andre78]

ahahaha, I didn't see that I can further scroll in that box. thanks

 

[senoj26]

I'm having issues with the console getting stuck at "Loading initial ramdisk" despite following the boot config update instructions. echo "simplefb" >> /etc/initramfs-tools/modules and update-initramfs -k all -u print nothing when run, but proxmox-boot-tool refresh gives me this output:

Running hook script 'proxmox-auto-removal'...
Running hook script 'zz-proxmox-boot'...
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
System booted in EFI mode but 'grub-efi-amd64' meta-package not installed!
Install 'grub-efi-amd64' to get updates.

---

Update: figured it out. I forgot to do the chroot again after reboot and as such, none of the changes I wrote about above were actually being applied to my system.