HowTo - an easy way to protect your PVE with encrypted storage

[MaLe]

Hi guys,

I want to share my solution that I'm using for encrypted storage on PVE. This manual is mainly addressed to beginners, because people who are familiar with ZFS know these things as natural features.

If you are using worlds best virtualization PVE in combination with worlds best storage technology ZFS, this is an easy way to protect your data and run your VMs on encrypted storage. This could be interesting specialy for cloudservers, because they are often running in difficult environments and for this solution, there is for example no need to give a password in early stage of boot. But also on premise it could be helpful. If someone steal your server, for these people it's not possible to access the data of your VMs or to start them. You always need the key to unlock the storage, like a closed door. And at the end, if you have a business and have to fulfill data protection laws, like German GDPR, it's often nessesary to store data of you clients in protected areas.

Please keep in mind: If you are on Linux, you know, there are always many different ways to get the same result. This is one way, it must not be the best but for me it works perfect. People who are using other filesystems could be satisfied with luks.

So let's start. These are the requirements:

- PVE6/7 environment
- ZFS storage

Step 1: Create encrypted filesystem on PVE's default ZFS pool "rpool"

zfs create -o encryption=on -o keyformat=passphrase rpool/encrypted_data

You will be prompted to give a password and to repeat it. Select a secure password with letters, numbers, special characters and with a propper lenght. Please take note, that loss of the password/key will be result in a LOSS OF ANY DATA on the encrypted storage. It's impossible to recover data without the key.

Step 2: Add filesystem as new pool to PVE storage

pvesm add zfspool local-crypt -pool rpool/encrypted_data

The new encrypted storage will be displayd as "local-crypt" in the UI and can now be used. If you are on a fresh PVE without any VMs and if you plan to use only encrypted storage, you could delete the default storage "local-zfs". With existing VMs it could be usefull to move them now to the encrypted storage.

Step 3: Mount the encrypted storage after reboots

zfs mount -l rpool/encrypted_data

For the reason, that mount of the encrypted storage needs a password, that cannot (should not) be done during start of the machine. So after every boot you have to mount the storage to start any VM located on this storage

Step 4: Make it easier to mount the storage after reboot

Add this line to ~/.bashrc:

alias start="zfs mount -l rpool/encrypted_data"

With this alias on the shell, even people who are using PVE with UI but not familar with the Linux shell are be able to easy start the storage after reboot. Simply type "start" at the shell, give password an everything works.

Lets have a look at some advantages and disadvanteges of this solution:

Advantages:
- VMs cannot be startet without knowing the key
- Key is not stored somewhere on the system
- Works "out of the box", no additional packages or products needed

Disadvanteges:
- VM Autostart will not work unless you hurry up to unlock the storage after reboot
- System need manual intervention after power loss
- Data will be lost FOREVER if key will be lost

BTW: You can do this also on PBS to have your backups placed on encrypted storage, even with separately encrypted storage for every PVE.

Please feel free to leave your toughts or recommendation.

Thanks,
Martin

 

[pickaboo]

Hi, nice tutorial, easily accessible and well described.

I face the problem of mounting the pools that are only data pools (not root system, just files) automatically.
It seems there are two possiblilities, one is via systemd, the other is with PAM.
https://talldanestale.dk/2020/04/06/zfs-and-homedir-encryption/

Using proxmox, we do not login with root at boot, so the PAM option is wrong.
The systemd option is mainly for root system.

I wonder if there is a way to use zfs pools mounted with passphrase at boot automatically.
Any idea?

In the documentation, they say there is a solution: https://pve.proxmox.com/wiki/ZFS_on_Linux#zfs_encryption
But does not give the details.
For LVM volume, we store the passphrase in /etc/cryptsetup, but with zfs, there is no clear way.

Finally, there is a huge tutorial on ArchWiki
https://wiki.archlinux.org/title/ZFS#Unlock_at_boot_time

All in all, I would like to have the option to automount the encrypted folder, but I don't really know how I should proceed.

 

[MaLe]

Hi,

many thanks for your reply. I never did further investigation in that direction because in our setup it is desired to manualy unlock the storage after reboot. We need this for data privacy reasons, to have the VMs in a non operational status in case of a theft. Additionaly I don't like to have key file or passphrase stored anywhere on the system.

On my older setup I used LUKS and EXT4 with a key file and automaticaly unlock during boot via /etc/fstab command line. Sorry, but for ZFS I can't help on that point.

 

[Dunuin]

Hi, nice tutorial, easily accessible and well described.

I face the problem of mounting the pools that are only data pools (not root system, just files) automatically.
It seems there are two possiblilities, one is via systemd, the other is with PAM.
https://talldanestale.dk/2020/04/06/zfs-and-homedir-encryption/

Using proxmox, we do not login with root at boot, so the PAM option is wrong.
The systemd option is mainly for root system.

I wonder if there is a way to use zfs pools mounted with passphrase at boot automatically.
Any idea?

In the documentation, they say there is a solution: https://pve.proxmox.com/wiki/ZFS_on_Linux#zfs_encryption
But does not give the details.
For LVM volume, we store the passphrase in /etc/cryptsetup, but with zfs, there is no clear way.

Finally, there is a huge tutorial on ArchWiki
https://wiki.archlinux.org/title/ZFS#Unlock_at_boot_time

All in all, I would like to have the option to automount the encrypted folder, but I don't really know how I should proceed.

I wrote this script to unlock passphrase encrypted ZFS pools/datasets after boot:

unlock_zfs.bash:

#!/bin/bash

DATASETS=( "yourPool/SomeDataset" "yourPool/AnotherDataset" )
PWDFILES=( "/path/to/key/SomeDataset.pwd" "/path/to/key/AnotherDataset.pwd" )

unlockDataset () {
    local dataset=$1
    local pwdfile=$2
    # check if dataset exists
    type=$(zfs get -H -o value type ${dataset})
    if [ "$type" == "filesystem" ]; then
        # check if dataset isn't already unlocked
        keystatus=$(zfs get -H -o value keystatus ${dataset})
        if [ "$keystatus" == "unavailable" ]; then
            zfs load-key ${dataset} < ${pwdfile}
            # check if dataset is now unlocked
            keystatus=$(zfs get -H -o value keystatus ${dataset})
            if [ "$keystatus" != "available" ]; then
                echo "Error: Unlocking dataset '${dataset}' failed"
                return 1
            fi
        else
            echo "Info: Dataset already unlocked"
            return 0
        fi
    else
        echo "Error: No valid dataset found"
        return 1
    fi
}

unlockAll () {
    local noerror=0
    # check if number of datasets and pwdfiles are equal
    if [ ${#DATASETS[@]} -eq ${#PWDFILES[@]} ]; then
        # loop though each dataset pwdfile pair
        for (( i=0; i<${#DATASETS[@]}; i++ )); do
            unlockDataset "${DATASETS[$i]}" "${PWDFILES[$i]}"
            if [ $? -ne 0 ]; then
                noerror=1
                break
            fi
        done
    else
        echo "Error: Wrong number of datasets/pwdfiles"
        noerror=1
    fi
    # mount all datasets
    if [ $noerror -eq 0 ]; then
        zfs mount -a
    fi
    return $noerror
}

unlockAll

The pwd files just contain the datasets passphrase, so the data isn't really safe when the passphares are stored unencrypted on the host. Because of that my whole PVE server uses full system encryption using LUKS (not described here...this is another big topic...) and I have to type in another passphrase via SSH (google for initramfs-dropbear) in order to be able unlock the LUKS and to boot PVE. Don't forget the chown root:root /path/to/key/SomeDataset.pwd && chmod 700 /path/to/key/SomeDataset.pwd so that only root is able to read the zfs passphrase.

And you might want to disable autostart of VMs/LXCs because that might fail when the ZFS pool ins't unlocked fast enough and write your own script to start the VMs/LXCs after the pool got unlocked.

So something like this:

pve_post_start.sh:

#!/bin/bash

echo "Starting up..."

# unlock encrypted datasets
bash /root/scripts/unlock_zfs.sh
sleep 10

######### Priority 1 #########

# start OPNsense VM to make DMZ/Intranet routing work
qm start 102
sleep 30

######### Priority 2 #########

# start PiHole LXC to make DynDNS/DNS work
pct start 116
sleep 20

# ...

echo "Startup finished"
exit 0

You can then add this as a systemd service so it will automatically unlock the ZFS datasets and start the VMs/LXCs after boot:

pve_post_start.service:

[Unit]
Description=PVE Start Script
Requires=network.target zfs.target local-fs.target remote-fs.target
After=pve-guests.service

[Service]
Type=oneshot
ExecStart=/bin/bash /root/scripts/pve_post_start.sh
User=root

[Install]
WantedBy=multi-user.target

Atleast here the above is working perfectly fine for 2 years now. Hope that might help someone, as it was a pain to find that out all on my own (and help of this forum).
I really hope the Proxmox Team will add official encryption features in the future so we don't need hacky ways like this any longer to encrypt our servers.


Also keep in mind that replication and migration won't work with encrypted datasets. So its not useful for clusters. See here: https://bugzilla.proxmox.com/show_bug.cgi?id=2350

 

[pickaboo]

Hi @Dunuin,

Thanks for that excellent tutorial and description.
The question of encryption is really crucial, complex and difficult to implement.
You did it great considering the constrains.
Best

 

[senorgeorgie]

Hi @MaLe,

Thank for this write-up. I have been looking for instructions like this for encrypting my Proxmox storage.
By following this guide are the additional storage drives add to the server also encrypted in the same filesystem or do those have to be mounted and encrypted as well?

 

[Dunuin]

That only encrypts a part of the system disk. Everything else isn't encrypted.

For encrypting a whole server I would recommend to to encrypt your root filesystem like described here:
https://forum.proxmox.com/threads/f...fs-using-proxmox-installer.127512/post-557792

Additional disks you could encrypt with ZFS native encryption or LUKS and for that using keyfiles.Those keyfiles you then could store on your encrypted root filesystem, so the other disks gets automatically unlocked too, once you unlock your root filesystem. See for example here: https://forum.proxmox.com/threads/f...fs-using-proxmox-installer.127512/post-557808

 

[senorgeorgie]

@Dunuin
Thank you, I appreciate the direction. I don't care for encrypting the whole server, just the storage. Your second link is what I needed.

 

[Dunuin]

Thank you, I appreciate the direction. I don't care for encrypting the whole server, just the storage. Your second link is what I needed.

Then keep in mind that you still might leak sensitive data as VMs/LXCsdata might be written unencrypted system disk's swap or log files.

 

[The Merchant]

I kept getting the following error: cannot create 'rpool/encrypted_data': no such pool 'rpool'
If this happens to you, use zpool create rpool /dev/{disk_name} to create the pool first, and then run the commands as provided by @MaLe