[TUTORIAL] Native full-disk encryption with ZFS

Bugbear

Member
Dec 29, 2020
37
6
13
Hello there!

I really like the idea of having the full root-filesystem encrypted using native ZFS encryption (only).
However, as I currently couldn't find any complete writeup on on this topic, here's what worked for me (with and without Secureboot on PVE 8.1):

1. Install with zfs (RAID0 for single-disk application)
2. Reboot into ISO > Advanced Options > Graphical, debug mode
3. Exit to bash with `exit` or Ctrl+D
4. Execute following commands:

Bash:
# Encrypt root dataset
zpool import -f rpool                        # Force import the ZFS pool named 'rpool'
zfs snapshot -r rpool/ROOT@copy              # Create a recursive snapshot of 'rpool/ROOT'
zfs send -R rpool/ROOT@copy | zfs receive rpool/copyroot            # Duplicate the snapshot to 'rpool/copyroot'
zfs destroy -r rpool/ROOT                    # Destroy the original 'rpool/ROOT' to replace it with an encrypted version
zfs create -o encryption=on -o keyformat=passphrase rpool/ROOT      # Create a new 'rpool/ROOT' with encryption
zfs send -R rpool/copyroot/pve-1@copy | zfs receive -o encryption=on rpool/ROOT/pve-1    # Restore 'pve-1' from the copy
zfs destroy -r rpool/copyroot                # Clean up by removing the temporary copy
zpool export rpool                           # Export the pool to finalize changes

# Prepare for chroot & destroy rpool/data dataset
zpool import -f -R /mnt rpool                # Import the pool with an alternate root at /mnt
zfs load-key -a                              # Load the encryption keys for all encrypted datasets
zfs destroy -r rpool/data                    # Destroy original dataset as after mounting pve-1 in the next step rpool/data will appear `busy` (see post #4 below)
zfs mount rpool/ROOT/pve-1                   # Mount the 'pve-1' dataset
mount -o rbind /proc /mnt/proc               # Recursively bind the /proc directory to the chroot environment
mount -o rbind /sys /mnt/sys                 # Recursively bind the /sys directory
mount -o rbind /dev /mnt/dev                 # Recursively bind the /dev directory
chroot /mnt /bin/bash                        # Change root into the new environment

# Create encrypt rpool/data dataset
dd if=/dev/urandom bs=32 count=1 of=/.data.key         # Create a new encryption key
chmod 400 /.data.key                                   # Set appropriate permissions for key
chattr +i /.data.key                                   # Make key immutable
zfs create -o encryption=on -o keylocation=file:///.data.key -o keyformat=raw rpool/data     # Create a new dataset with encryption enabled
# Setup systemd service for automatic unlocking of rpool/data on boot
sudo cat > /etc/systemd/system/zfs-load-key.service <<'EOF'
[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a

[Install]
WantedBy=zfs-mount.service
EOF
systemctl enable zfs-load-key

## Optional! Only needed if stuck at boot with ZFS-encryption enabled (see post #3 below): Update boot configuration
echo "simplefb" >> /etc/initramfs-tools/modules       # Add 'simplefb' to initramfs modules
update-initramfs -k all -u                            # Update all initramfs images
proxmox-boot-tool refresh                             # Refresh Proxmox boot configuration to apply changes

# Cleanup and reboot
exit
umount /mnt/proc                              # Unmount /proc
umount /mnt/sys                               # Unmount /sys
umount /mnt/dev                               # Unmount /dev (if target is busy, check for nested mounts)
zfs unmount rpool/data                  # Unmount the ZFS dataset
zfs unmount rpool/ROOT/pve-1                  # Unmount the ZFS dataset
zpool export rpool                            # Export the ZFS pool
Ctrl + Alt + Del                              # Use key combination to reboot the system

----
Credits to:
@nschemel https://forum.proxmox.com/threads/encrypting-proxmox-ve-best-methods.88191/#post-387731
@bindi https://forum.proxmox.com/threads/unable-to-unlock-zfs-root-dataset-during-boot.138172/post-616393
@Stoiko Ivanov https://forum.proxmox.com/threads/e...option-system-cannot-boot.104377/#post-450093
@yvesh https://gist.github.com/yvesh/ae77a68414484c8c79da03c4a4f6fd55
@Tommy Tran on PrivSec https://privsec.dev/posts/linux/using-native-zfs-encryption-with-proxmox/

PS: Search engines aren't what they used to be :/ ...
 
Last edited:
You forgot to encrypt the "data" dataset. So VMs/LXCs stored on your disk won't be encrypted.

Why is the "simplefb" required? As far as I remember I never did that when encrypting my rpool. I only read about "simplefb" in context of GPU passthrough.

In case you want to be able to unlock it headless I can highly recommend using the dropbear-initramfs SSH server.
 
Last edited:
  • Like
Reactions: semanticbeeng
Hey @Dunuin , thank you very much for the feedback! Dropbear is truely neat (and easy to set up with PrivSecs guide from above), personally I'll just use an IP-KVM.
Good catch with rpool/data! Totally forgot that ... I just added it to the guide.

As for the simplefb-module I wondered too, however without this solution on boot I get stuck at this screen (like @jwalker in this thread) with SecureBoot disabled:
1000006347.jpg
and stuck at this screen with SecureBoot enabled:
pve.jpg
However, while being stuck at these screens in the console, I can still unlock the zfs dataset with dropbear and then the OS loads perfectly fine (on the console too!).
So it's "just" a graphical issue with this message hanging on boot, fixable with the simplefb-module in the initramfs.

Cheers!
 
Last edited:
I followed these steps exactly and managed to get ROOT encrypted. Unfortunately, I ran into an error when trying to encrypt rpool/data. I made it as far as "zfs destroy -r rpool/data". This resulted in the error "cannot destroy rpool/data: dataset is busy". Not sure where to go from here.

PVE 8.1 w/ secure boot, x2 nvme zfs raid 1
 
Hi @dynostatic, I'm sorry for this issue. Please try the "Encrypt rpool/data" section before the "Prepare for chroot" section and thus before importing the pool.

EDIT: I just updated the guide so this workaround won't be needed anymore. Should work flawlessly now.
 
Last edited:
Thank you @Bugbear for the instructions, I was able to set up a Proxmox host at at my hosting provider.

However, when trying to create a LXC, I am getting this error:

TASK ERROR: unable to create CT 100 - zfs error: cannot create 'rpool/data/subvol-100-disk-0': encryption root's key is not loaded or provided

Do you have a suggestion how to get around this?

Thank you!
 
Hi @andre78, you need to catch up on the # Setup systemd service for automatic unlocking of rpool/data on boot section for automatic unlocking of rpool/data on boot. Otherwise you'd need to execute `zfs load-key -a` on every reboot.
 
Last edited:
I'm having issues with the console getting stuck at "Loading initial ramdisk" despite following the boot config update instructions. echo "simplefb" >> /etc/initramfs-tools/modules and update-initramfs -k all -u print nothing when run, but proxmox-boot-tool refresh gives me this output:

Code:
Running hook script 'proxmox-auto-removal'...
Running hook script 'zz-proxmox-boot'...
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
System booted in EFI mode but 'grub-efi-amd64' meta-package not installed!
Install 'grub-efi-amd64' to get updates.



Update: figured it out. I forgot to do the chroot again after reboot and as such, none of the changes I wrote about above were actually being applied to my system.
 
Last edited:
Great post -- thank-you for taking the time for this. After a couple attempts, I got it first try (LEGO movie reference).

I did have some questions.

1. When issuing the command "zfs create -o encryption=on -o keyformat=passphrase rpool/ROOT" -- wouldn't something like "zfs create -o encryption=aes-256-gcm -o keyformat=passphrase rpool/ROOT" be more secure? (or, does Debian default to this anyway?)

2. Similar question to the aforementioned with "zfs create -o encryption=on -o keylocation=file:///.data.key -o keyformat=raw rpool/data" -- I assume this inherits the encryption algorithm used for the parent?

3-A - When issuing the "dd if=/dev/urandom bs=32 count=1 of=/.data.key" command -- why bs=32? Why not bs=512? Is there a limit that can be used here?

32 is the maximum raw, apparently.

3-B - Does the name .data.key have to be used? Or can it be anything?

It looks like it can be anything.

4. I would (likely) end up with another pool, creatively named "tank" which would hold the bulk of my data. I don't want to use a key file, but, I'm also lazy and would like to reuse the same passphrase I used for root. Is there a way to extend the boot prompt to unlock both root, AND tank at the same time?

Thank-you!
 
Last edited:
Great post -- thank-you for taking the time for this. After a couple attempts, I got it first try (LEGO movie reference).

I did have some questions.

1. When issuing the command "zfs create -o encryption=on -o keyformat=passphrase rpool/ROOT" -- wouldn't something like "zfs create -o encryption=aes-256-gcm -o keyformat=passphrase rpool/ROOT" be more secure? (or, does Debian default to this anyway?)

2. Similar question to the aforementioned with "zfs create -o encryption=on -o keylocation=file:///.data.key -o keyformat=raw rpool/data" -- I assume this inherits the encryption algorithm used for the parent?

3-A - When issuing the "dd if=/dev/urandom bs=32 count=1 of=/.data.key" command -- why bs=32? Why not bs=512? Is there a limit that can be used here?

32 is the maximum raw, apparently.

3-B - Does the name .data.key have to be used? Or can it be anything?

It looks like it can be anything.

4. I would (likely) end up with another pool, creatively named "tank" which would hold the bulk of my data. I don't want to use a key file, but, I'm also lazy and would like to reuse the same passphrase I used for root. Is there a way to extend the boot prompt to unlock both root, AND tank at the same time?

Thank-you!

Not OP but a few thoughts on your questions:

1. I don't know what the default encryption scheme is so I went with "aes-256-gcm" everywhere. No reason not to anyway, and now I know for sure it's using the cipher I want.

2. Again, I'm not sure but best to just go on the safe side and explicitly write the cipher

4. Why don't you want to use a key file?
For my additional pool I simply followed the same steps as done for rpool/data. Create a random keyfile, create your pool with that key, then create a new service to load it automatically. Basically just duplicate what is already happening for rpool/data for your own pool.
Then when you type in the passphrase for rpool/ROOT, the key load process happens for rpool/data and your new pool, which would be the same as reusing the passphrase (at least as far as user interaction is concerned).
 
  • Like
Reactions: np247
Not OP but a few thoughts on your questions:

1. I don't know what the default encryption scheme is so I went with "aes-256-gcm" everywhere. No reason not to anyway, and now I know for sure it's using the cipher I want.

2. Again, I'm not sure but best to just go on the safe side and explicitly write the cipher

4. Why don't you want to use a key file?
For my additional pool I simply followed the same steps as done for rpool/data. Create a random keyfile, create your pool with that key, then create a new service to load it automatically. Basically just duplicate what is already happening for rpool/data for your own pool.
Then when you type in the passphrase for rpool/ROOT, the key load process happens for rpool/data and your new pool, which would be the same as reusing the passphrase (at least as far as user interaction is concerned).
Thank-you for this!

4. Good idea. My opposition is I'm fairly lazy/stupid, and will almost certainly lose said keyfile and be screwed in the future.

I wonder how difficult it would be to add dropbear support to the aforementioned guide. I like your suggestion -- but that initial password is the kicker for me. I want to use the maximum passphrase possible (I believe 512 characters), which I would copy and paste. However, my Supermicro IPMI doesn't allow for copy and paste.

A compromise might be to stick with the initial passphrase/keyfile for the Promox installation, then use my passphrase
 
The commands I ended up using for encrypting "tank" were as follows:

Bash:
dd if=/dev/urandom bs=32 count=1 of=/.tank.key
chmod 400 /.tank.key
chattr +i /.tank.key                 
zpool create -o ashift=12  -O atime=off -O compression=lz4 -o autoexpand=on -O mountpoint=none tank mirror /dev/sdb /dev/sdc
zfs create -o atime=off -o compression=lz4 -o encryption=aes-256-gcm -o keylocation=file:///.tank.key -o keyformat=raw -o mountpoint=/mnt/tank tank/crypt

I didn't have to create a secondary script or anything -- rebooting, re-authenticating to unlock rpool/ROOT was sufficient to automagically unlock both key file encrypted volumes.

It's probably more logical to just re-use the existing .data.key, but meh, whatever.
 
Last edited:
Thanks for this wonderful guide.
I've successfully installed PVE 8.2 and, doing some small modification, completed ZFS encryption with remote key location (http).
Here are steps i've done.
Setup networking in rescue environment, and execute apt update && apt install libcurl4 -y ; this dependency is needed from zfs create command in order to fetch remote http data.
Add keylocation=http://192.168.1.50/your-remote-zfs-password-file keyformat=passphrase to both zfs create commands
Add IP=192.168.1.100::192.168.1.1:255.255.255.0::eno1:off to /etc/initramfs-tools/initramfs.conf inside chroot
Execute following snippet inside chroot to create configure_networking hook script, needed to bring up networking in initrd
Bash:
echo '#!/bin/sh
if [ "$1" = "prereqs" ] ; then echo "";exit 0;fi
. /scripts/functions
configure_networking
' > /etc/initramfs-tools/scripts/local-top/configure_networking
chmod +x  /etc/initramfs-tools/scripts/local-top/configure_networking
Regenerate initramfs, as per guide.
Please replace 192.168.1.100 with your PVE IP, 192.168.1.1 with your gateway, 255.255.255.0 with your netmask, eno1 with your interface name.

Last note: i've noticed that rpool/var-lib-vz was not encrypted; i've deleted and recreated it whith keylocation/keyformat parameters, setting its mountpoint to /var/lib/vz
 
Last edited:
Our server is located in the office. To ensure the data is protected even in the event of theft, it is not an option to store the key on an unencrypted pool. Presenting a passphrase or key file, for example via Dropbear, during or after booting, is also not ideal, as it requires IT expertise, which is not always available.

My preferred solution would be to have a key file on a USB stick. If the USB stick is plugged in, the key file can be loaded, and the unlock would happen automatically without any further input. If the USB stick is not plugged in, it should be possible to start the unlock manually via SSH. Unfortunately, I haven’t been able to get this USB idea to work. Does anyone else have this or a similar use case, and how was it solved?
 
  • Like
Reactions: fpausp
Hello there!

I really like the idea of having the full root-filesystem encrypted using native ZFS encryption (only).
However, as I currently couldn't find any complete writeup on on this topic, here's what worked for me (with and without Secureboot on PVE 8.1):

1. Install with zfs (RAID0 for single-disk application)
2. Reboot into ISO > Advanced Options > Graphical, debug mode
3. Exit to bash with `exit` or Ctrl+D
4. Execute following commands:

Bash:
# Encrypt root dataset
zpool import -f rpool                        # Force import the ZFS pool named 'rpool'
zfs snapshot -r rpool/ROOT@copy              # Create a recursive snapshot of 'rpool/ROOT'
zfs send -R rpool/ROOT@copy | zfs receive rpool/copyroot            # Duplicate the snapshot to 'rpool/copyroot'
zfs destroy -r rpool/ROOT                    # Destroy the original 'rpool/ROOT' to replace it with an encrypted version
zfs create -o encryption=on -o keyformat=passphrase rpool/ROOT      # Create a new 'rpool/ROOT' with encryption
zfs send -R rpool/copyroot/pve-1@copy | zfs receive -o encryption=on rpool/ROOT/pve-1    # Restore 'pve-1' from the copy
zfs destroy -r rpool/copyroot                # Clean up by removing the temporary copy
zpool export rpool                           # Export the pool to finalize changes

# Prepare for chroot & destroy rpool/data dataset
zpool import -f -R /mnt rpool                # Import the pool with an alternate root at /mnt
zfs load-key -a                              # Load the encryption keys for all encrypted datasets
zfs destroy -r rpool/data                    # Destroy original dataset as after mounting pve-1 in the next step rpool/data will appear `busy` (see post #4 below)
zfs mount rpool/ROOT/pve-1                   # Mount the 'pve-1' dataset
mount -o rbind /proc /mnt/proc               # Recursively bind the /proc directory to the chroot environment
mount -o rbind /sys /mnt/sys                 # Recursively bind the /sys directory
mount -o rbind /dev /mnt/dev                 # Recursively bind the /dev directory
chroot /mnt /bin/bash                        # Change root into the new environment

# Create encrypt rpool/data dataset
dd if=/dev/urandom bs=32 count=1 of=/.data.key         # Create a new encryption key
chmod 400 /.data.key                                   # Set appropriate permissions for key
chattr +i /.data.key                                   # Make key immutable
zfs create -o encryption=on -o keylocation=file:///.data.key -o keyformat=raw rpool/data     # Create a new dataset with encryption enabled
# Setup systemd service for automatic unlocking of rpool/data on boot
sudo cat > /etc/systemd/system/zfs-load-key.service <<'EOF'
[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a

[Install]
WantedBy=zfs-mount.service
EOF
systemctl enable zfs-load-key

## Optional! Only needed if stuck at boot with ZFS-encryption enabled (see post #3 below): Update boot configuration
echo "simplefb" >> /etc/initramfs-tools/modules       # Add 'simplefb' to initramfs modules
update-initramfs -k all -u                            # Update all initramfs images
proxmox-boot-tool refresh                             # Refresh Proxmox boot configuration to apply changes

# Cleanup and reboot
exit
umount /mnt/proc                              # Unmount /proc
umount /mnt/sys                               # Unmount /sys
umount /mnt/dev                               # Unmount /dev (if target is busy, check for nested mounts)
zfs unmount rpool/data                  # Unmount the ZFS dataset
zfs unmount rpool/ROOT/pve-1                  # Unmount the ZFS dataset
zpool export rpool                            # Export the ZFS pool
Ctrl + Alt + Del                              # Use key combination to reboot the system

----
Credits to:
@nschemel https://forum.proxmox.com/threads/encrypting-proxmox-ve-best-methods.88191/#post-387731
@bindi https://forum.proxmox.com/threads/unable-to-unlock-zfs-root-dataset-during-boot.138172/post-616393
@Stoiko Ivanov https://forum.proxmox.com/threads/e...option-system-cannot-boot.104377/#post-450093
@yvesh https://gist.github.com/yvesh/ae77a68414484c8c79da03c4a4f6fd55
@Tommy Tran on PrivSec https://privsec.dev/posts/linux/using-native-zfs-encryption-with-proxmox/

PS: Search engines aren't what they used to be :/ ...
Hi,

Thanks you so much for the guide, this is what I was looking for.
Unfortunately, it does not work on a fresh 8.2 installation. I tried 2 times with two reinstalls.

After the first encryption command I have an error saying it cannot be mointed (photo attached).
If I ignore and continue, I have then another error and cannot continue.

I have no idea on how I can solve this. Maybe I will try downgrade to Proxmox 8.1 and once encrypted try an upgrade from there.

Any help will be well appreciated !
 

Attachments

  • 1000004855.jpg
    1000004855.jpg
    511.2 KB · Views: 4
  • 1000004856.jpg
    1000004856.jpg
    353.4 KB · Views: 4
That works beautifully, but I would like to have the first passphrase stored in TPM2.0 - could that be achieved too? Is this possible and if so - how? Btw. the only caveat is that eventhough having put simplefb in the file, rebuilt initramfs I still get same as the one guy on the third post..

Maybe even eliminate the need of grub - just tpm2.0 stored key -> boot from efi ?
 
Last edited:
That works beautifully, but I would like to have the first passphrase stored in TPM2.0 - could that be achieved too? Is this possible and if so - how? Btw. the only caveat is that eventhough having put simplefb in the file, rebuilt initramfs I still get same as the one guy on the third post..

Maybe even eliminate the need of grub - just tpm2.0 stored key -> boot from efi ?
Sorry I don't know. Just keep in mind that what you described is useful only to protect your data if the disk is lost, sold or stolen in the future. Because if your server/computer/Nas/whatever system you have is stolen, your data will be decrypted automatically by the bad actor using the TMP.
It is the same problem with Microsoft BitLocker, this is why multiple companies enforce the use of a mandatory PIN code.

Hi,

Thanks you so much for the guide, this is what I was looking for.
Unfortunately, it does not work on a fresh 8.2 installation. I tried 2 times with two reinstalls.

After the first encryption command I have an error saying it cannot be mointed (photo attached).
If I ignore and continue, I have then another error and cannot continue.

I have no idea on how I can solve this. Maybe I will try downgrade to Proxmox 8.1 and once encrypted try an upgrade from there.

Any help will be well appreciated !
For the follow up, I managed to encrypt the 8.2 installation after many tries and errors.
Bash:
# Source : https://forum.proxmox.com/threads/full-disk-encryption-with-zfs-using-proxmox-installer.127512/post-557792
# Import ZFS pool
zpool import -f rpool

# Source : https://forum.proxmox.com/threads/full-disk-encryption-with-zfs-using-proxmox-installer.127512/post-557808
zpool set autotrim=on rpool
zfs set relatime=on rpool
# Configure the pool
zfs set compression=lz4 rpool
zfs set recordsize=128k rpool
zfs set acltype=posix rpool
zfs set checksum=blake3 rpool
zfs set dnodesize=auto rpool
zfs set dedup=blake3,verify rpool
zfs set xattr=sa rpool

# Snapshot rpool/ROOT
zfs snapshot -r rpool/ROOT@copy
# Create copy of unencrypted rpool/ROOT and all childs
zfs send -R rpool/ROOT@copy | zfs recv rpool/copyroot
# Destroy unencrypted rpool/ROOT
zfs destroy -r rpool/ROOT
# Create new encrypted rpool/ROOT
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase rpool/ROOT
# Copy and encrypt unencrypted rpool/copyroot/pve-1
zfs send -R rpool/copyroot/pve-1@copy | zfs recv -o encryption=on rpool/ROOT/pve-1
# Destroy copy
zfs destroy -r rpool/copyroot
# Destroy snapshots
zfs destroy rpool/ROOT/pve-1@copy
# Apply quota to use around 90% of disk.
zfs set quota=1.6T rpool/ROOT
zfs set quota=1.6T rpool/ROOT/pve-1

# List all with encryption status
zfs get encryption
# rpool/ROOT/* is encrypted. rpool/var-lib-vz is NOT
# Export pool
zpool export rpool
reboot

After reboot, encrypt vz folder:
Bash:
### After rebooting, on PVE, not its ISO or rescue mode
### NOW recreating dataset /var-lib-vz with mountpoint in /var/lib/vz

# Create long random key (diceware passphrase)
# nano /.data.key
# Remove all but ASCII characters
# perl -i -pe 's/[^ -~]//g' /.data.key
# OR use urandom to generate a string random of 32 chars.
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1 > /.data.key
# Set the appropriate permission
chmod 400 /.data.key
# Make the key immutable
chattr +i /.data.key
## WARNING WARNING WARNING - Backup file content NOW!
# Destroy unencrypted rpool/var-lib-vz (no content as this is an after install step)
umount /var/lib/vz
# Data should be empty so you could destroy it
zfs destroy -r rpool/var-lib-vz
# Create encrypted zfs
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=file:///.data.key rpool/var-lib-vz
zfs set mountpoint=/var/lib/vz rpool/var-lib-vz
# Apply quota
zfs set quota=500G rpool/var-lib-vz

# Get encryption status : everything should be encrypted except the pool (rpool)
zfs get encryption

## DO THE SAME FOR rpool/data,
## this is used by default by proxmox for lxc

zfs destroy -r rpool/data
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=file:///.data.key rpool/data
zfs set mountpoint=/rpool/data rpool/data
# Apply quota
zfs set quota=500G rpool/data

For auto decryption on boot (for data and vz):

Bash:
# nano /etc/systemd/system/zfs-load-key.service
[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a

[Install]
WantedBy=zfs-mount.service

Activate the service:
Bash:
systemctl daemon-reload
systemctl enable zfs-load-key