Encrypting Proxmox VE (Best Methods)

[Whitterquick]

I have been looking for the best way to encrypt the Proxmox boot drive (one that is not using a hardware RAID setup). From what I have seen the most common way of doing this is over a Debian install with LUKS encryption. I have also found this method. Is there any advantage using one method over another?

Also is there any way of encrypting the VM storage drive/volume in a non-RAID setup? Is the preferred method to individually encrypt each VM when setting them up rather than the drive that they are stored on?

I have not been able to find the answer to these questions so hopefully someone can answer. Thanks.

 

[oguz]

hi,

Is there any advantage using one method over another?

the method you linked does the encryption on an already existing proxmox installation (with default settings). in that case you don't need to start from a debian installation.
though those instructions are from 2016 and they won't work as is anymore (at least regarding the name and mountpoint of the data lv)

if you have good amount of ram, you can also use encrypted datasets with zfs [0], this works well without hardware RAID and you get the other benefits of ZFS.

Also is there any way of encrypting the VM storage drive/volume in a non-RAID setup? Is the preferred method to individually encrypt each VM when setting them up rather than the drive that they are stored on?

if you use encrypted zfs dataset for your VM storage, then the whole drive will be encrypted.

otherwise you can install cryptsetup and setup LUKS inside each VM, or encrypt the LV they're being stored on the host.

obviously this all means you'll have to type the decryption password during boot if the host root disk is encrypted. also the encrypted volumes need to be decrypted before guests are started at boot (so this can inhibit your ability to automatically start guests at boot). if you still want that functionality you can keep an unencrypted volume for those VMs that should start at boot.

hope this helps.

[0]: https://pve.proxmox.com/wiki/ZFS_on_Linux#zfs_encryption

 

[Whitterquick]

hi,


the method you linked does the encryption on an already existing proxmox installation (with default settings). in that case you don't need to start from a debian installation.
though those instructions are from 2016 and they won't work as is anymore (at least regarding the name and mountpoint of the data lv)

Thank you Oguz. Is there an updated guide or instructions on how to encrypt an existing Proxmox installation?

 

[H4R0]

Thank you Oguz. Is there an updated guide or instructions on how to encrypt an existing Proxmox installation?

If you want to go with zfs you can follow my write up https://herold.space/proxmox-zfs-full-disk-encryption-with-ssh-remote-unlock/

 

[Dunuin]

I installed Debian on an LUKS encrypted mdraid software raid array (works without raid too). Google for initramfs-dropbear. With that you can unlock your encrypted system partition using SSH so you don't need to manually type in the password and use password managers.
The VMs are stored on encrypted ZFS pools and I wrote a script that auto unlocks the ZFS pools after boot. The keys for the ZFS pools are stored on the unlocked LUKS system partition.
So everything I need to do is to power on the server, login using SSH with RSA key and paste my LUKS passphrase. Then everything boots up, unlocks and the VMs are starting automatically.

 

[Whitterquick]

I installed Debian on an LUKS encrypted mdraid software raid array (works without raid too). Google for initramfs-dropbear. With that you can unlock your encrypted system partition using SSH so you don't need to manually type in the password and use password managers.
The VMs are stored on encrypted ZFS pools and I wrote a script that auto unlocks the ZFS pools after boot. The keys for the ZFS pools are stored on the unlocked LUKS system partition.
So everything I need to do is to power on the server, login using SSH with RSA key and paste my LUKS passphrase. Then everything boots up, unlocks and the VMs are starting automatically.

Is GRUB on encrypted LUKS?

 

[Dunuin]

No, GRUB or ESP needs to be unencrypted as far as I know. The same for the boot partition. Grub/ESP boots the unencrypted initramfs from the boot partition, this asks for the LUKS password and only if you unlocked the root partition proxmox is able to start.

Here it looks like this:

able law.
root@Hypervisor:~# lsblk
NAME                 MAJ:MIN  RM   SIZE RO TYPE  MOUNTPOINT
sda                    8:0     0  93.2G  0 disk
├─sda1                 8:1     0     1M  0 part
├─sda2                 8:2     0   286M  0 part  /boot/efi
├─sda3                 8:3     0   488M  0 part
│ └─md0                9:0     0   487M  0 raid1 /boot
└─sda4                 8:4     0  92.4G  0 part
  └─md1                9:1     0  92.3G  0 raid1
    └─md1_crypt      253:0     0  92.3G  0 crypt
      ├─vgpmx-lvroot 253:1     0    21G  0 lvm   /
      └─vgpmx-lvswap 253:2     0    61G  0 lvm   [SWAP]
sdb                    8:16    0  93.2G  0 disk
├─sdb1                 8:17    0     1M  0 part
├─sdb2                 8:18    0   286M  0 part
├─sdb3                 8:19    0   488M  0 part
│ └─md0                9:0     0   487M  0 raid1 /boot
└─sdb4                 8:20    0  92.4G  0 part
  └─md1                9:1     0  92.3G  0 raid1
    └─md1_crypt      253:0     0  92.3G  0 crypt
      ├─vgpmx-lvroot 253:1     0    21G  0 lvm   /
      └─vgpmx-lvswap 253:2     0    61G  0 lvm   [SWAP]
sdc                    8:32    0 186.3G  0 disk
├─sdc1                 8:33    0 186.3G  0 part
└─sdc9                 8:41    0     8M  0 part
sdd                    8:48    0 186.3G  0 disk
├─sdd1                 8:49    0 186.3G  0 part
└─sdd9                 8:57    0     8M  0 part
sde                    8:64    0 186.3G  0 disk
├─sde1                 8:65    0 186.3G  0 part
└─sde9                 8:73    0     8M  0 part
sdf                    8:80    0 186.3G  0 disk
├─sdf1                 8:81    0 186.3G  0 part
└─sdf9                 8:89    0     8M  0 part
sdg                    8:96    0 186.3G  0 disk
├─sdg1                 8:97    0 186.3G  0 part
└─sdg9                 8:105   0     8M  0 part
sdh                    8:112   0   2.7T  0 disk
├─sdh1                 8:113   0   2.7T  0 part
└─sdh9                 8:121   0     8M  0 part
sdi                    8:128   0   2.7T  0 disk
├─sdi1                 8:129   0   2.7T  0 part
└─sdi9                 8:137   0     8M  0 part

Sdc to sdg is a ZFS pool with a encrypted dataset storing all the VMs.
Sdh and sdi is another ZFS pool.
Sda and sdb are my system drives. Partition 2 is for the ESP, but I used GRUB not UEFI, so it isn't used. Partition 3 is the boot partition which needs to be unencrypted. Partition 4 is LUKS encrypted LVM with a root LV and a swap LV.

 

[nschemel]

As I've a fully encrypted Ubuntu 20.04 with zfs running, I wanted to test if the same would be possible for proxmox, because I want to migrate the ubuntu installation over to proxmox.

Im not the biggest fan of the luks + zfs version, because of the added complexity, while it should be possible only with zfs. So I searched a while and found some bits and pieces her in the forums and the zfs repo.

Right now I have the root of my proxmox server encrypted and the password is asked at boot via console and ssh. So I can input the password locally or remotely.

Because I found no complete tutorial, I created a forum account to share with you what I've done so far.

Warning: You will need zfs 0.8.5 or newer, as far as I know. ,So this will only work proxmox 6.3 or newer. I've tested it with the newly released 6.4 version.

1. Install proxmox and choose zfs raid 0 in the installer.
2. After installation I booted right into the rescue mode and followed this link to a gist with instructions(which is originally from this forum. Thanks @Yakuraku)
   • In addition to the tutorial, I had to move the mount point of the dataset "copyroot/pve-1" to /rpool/copyroot/pve-1 to get work. Because without this, there where two datasets with the same mount point and the system didn't boot.
   • Another way would be to delete delete the old copyroot stuff after migrating. That's what you would like to do anyways.
3. After that I was able to boot into the system.
4. Because I want to unlock my encrypted root dataset remotely, there was one other step to do. Since zfs 0.8.5 it's possible to unlock zfs datasets via ssh while the system is booting. It's fairly simple, but a bit harder to find. Just follow the instructions at the bottom of the initramfs readme from zfs github repo
5. After that there was only one more step to do: encrypting the dataset for vms and containers. That's well documented in the proxmox documentation
   1. Create an encryption key:

      openssl rand -hex -out /root/data-encrypted.key 32

   2. Create new encrypted dataset:

      zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///root/data-encrypted.key rpool/data/encrypted

   3. Add the encrypted dataset to proxmox:

      pvesm add zfspool encrypted_zfs -pool rpool/data/encrypted

I hope that little writeup helps anyone.

 

[Dunuin]

Thanks, good to know. When I installed my Proxmox it was using ZFS 0.8.3 and it wasn't possible to boot from a encrypted root.

You wrote "Install proxmox and choose zfs raid 0 in the installer". What about Raid1? Will mirroring work too? And what about GRUB (or is only ESP with UEFI possible)? What about a encrypted SWAP partition?

 

[nschemel]

Thanks, good to know. When I installed my Proxmox it was using ZFS 0.8.3 and it wasn't possible to boot from a encrypted root.

You wrote "Install proxmox and choose zfs raid 0 in the installer". What about Raid1? Will mirroring work too? And what about GRUB (or is only ESP with UEFI possible)? What about a encrypted SWAP partition?

I choose raid 0, because I only have one drive and that's the only option to install proxmox on zfs. I think all zfs raid options should work. As far as I understand only ESP with UEFI will work, because grub only supports a subset of zfs options.

Regarding encrypted swap I didn't look at it so far.

 

[Whitterquick]

If installing Proxmox over an encrypted Debian, is it still possible to use an encrypted separate physical hard drive for storing VMs?

 

[Dunuin]

If installing Proxmox over an encrypted Debian, is it still possible to use an encrypted separate physical hard drive for storing VMs?

Yes, thats the easy part.

 

[Whitterquick]

Yes, thats the easy part.

Is the hard part unlocking it or what?

When selecting the hard drive that the VMs will be stored on, the encrypted one is nowhere to be seen! I don’t have any issue unlocking via cli as I don’t expect the server to be turned off and on all the time, I just need to know the process.

 

[Dunuin]

Is the hard part unlocking it or what?

When selecting the hard drive that the VMs will be stored on, the encrypted one is nowhere to be seen! I don’t have any issue unlocking via cli as I don’t expect the server to be turned off and on all the time, I just need to know the process.

The hard part is to get Proxmox boot from an encrypted root filesystem.

Do you use ZFS native encryption or LUKS with LVM?
Unlocking and mounting is a little bit annoying if you use ZFS native encryption without a keyfile but with a passphrase because you will need to manually type in the passphrase to unlock it or write your own script so the passphrase is piped in as user input.

 

[Whitterquick]

The hard part is to get Proxmox boot from an encrypted root filesystem.

Do you use ZFS native encryption or LUKS with LVM?
Unlocking and mounting is a little bit annoying if you use ZFS native encryption without a keyfile but with a passphrase because you will need to manually type in the passphrase to unlock it or write your own script so the passphrase is piped in as user input.

Still testing, but LUKS for now. I created an encrypted Debian VM and installed Proxmox over it successfully as per wiki. I can start the VM but cannot connect to the web interface and cant ping it either. Wondering what I am doing wrong...

 

[misku]

Just FYI, there's already a feature request for Proxmox ZFS native encryption in the installer. Votes are kindly appreciated

 

[9bitbyte]

As I've a fully encrypted Ubuntu 20.04 with zfs running, I wanted to test if the same would be possible for proxmox, because I want to migrate the ubuntu installation over to proxmox.

Im not the biggest fan of the luks + zfs version, because of the added complexity, while it should be possible only with zfs. So I searched a while and found some bits and pieces her in the forums and the zfs repo.

Right now I have the root of my proxmox server encrypted and the password is asked at boot via console and ssh. So I can input the password locally or remotely.

Because I found no complete tutorial, I created a forum account to share with you what I've done so far.

Warning: You will need zfs 0.8.5 or newer, as far as I know. ,So this will only work proxmox 6.3 or newer. I've tested it with the newly released 6.4 version.

1. Install proxmox and choose zfs raid 0 in the installer.
2. After installation I booted right into the rescue mode and followed this link to a gist with instructions(which is originally from this forum. Thanks @Yakuraku)
   • In addition to the tutorial, I had to move the mount point of the dataset "copyroot/pve-1" to /rpool/copyroot/pve-1 to get work. Because without this, there where two datasets with the same mount point and the system didn't boot.
   • Another way would be to delete delete the old copyroot stuff after migrating. That's what you would like to do anyways.
3. After that I was able to boot into the system.
4. Because I want to unlock my encrypted root dataset remotely, there was one other step to do. Since zfs 0.8.5 it's possible to unlock zfs datasets via ssh while the system is booting. It's fairly simple, but a bit harder to find. Just follow the instructions at the bottom of the initramfs readme from zfs github repo
5. After that there was only one more step to do: encrypting the dataset for vms and containers. That's well documented in the proxmox documentation
   1. Create an encryption key:

      openssl rand -hex -out /root/data-encrypted.key 32

   2. Create new encrypted dataset:

      zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///root/data-encrypted.key rpool/data/encrypted

   3. Add the encrypted dataset to proxmox:

      pvesm add zfspool encrypted_zfs -pool rpool/data/encrypted

I hope that little writeup helps anyone.

Do you perhaps know if this process is SAFE for an existing/running system with 2xdisk zfs mirror rpool (compression is enabled too)?

And also, what happens to rpool/data? Is it just left unencrypted? That means I would have to encrypt it separately?

 

[Dunuin]

I don't see the point in adding encryption later. Atleast not for SSDs. All SSDs got some spare area that you can't access but where your SSD will store your data. So lets say every SSD is actually 10% bigger than sold as (used for wear leveling and caching), so a 1000 GB SSD actually got 1100GB of storage but where 100GB aren't accessible. If you now wipe that SSD to delete old unencrypted data or if your overwrite 1TB of unencrypted data with encrypted data, there is still up to 100GB of unencrypted data remaining in the unaccessible area which you can't overwrite but could contain your clear text passwords or other sensitive data that still would remain on your "fully" encrypted drive. If you really want to get rid of all your old unencrypted data you need to tell the SSDs firmware to do a "secure erase" where everything will be set back to factory defaults which should include deleting the complete 1100GB including this spare area that you can't overwrite otherwise. Only if you know there will be no unencrypted data left on the drive it makes sense to actually encrypt it.

 

[Whitterquick]

I don't see the point in adding encryption later. Atleast not for SSDs. All SSDs got some spare area that you can't access but where your SSD will store your data. So lets say every SSD is actually 10% bigger than sold as (used for wear leveling and caching), so a 1000 GB SSD actually got 1100GB of storage but where 100GB aren't accessible. If you now wipe that SSD to delete old unencrypted data or if your overwrite 1TB of unencrypted data with encrypted data, there is still up to 100GB of unencrypted data remaining in the unaccessible area which you can't overwrite but could contain your clear text passwords or other sensitive data that still would remain on your "fully" encrypted drive. If you really want to get rid of all your old unencrypted data you need to tell the SSDs firmware to do a "secure erase" where everything will be set back to factory defaults which should include deleting the complete 1100GB including this spare area that you can't overwrite otherwise. Only if you know there will be no unencrypted data left on the drive it makes sense to actually encrypt it.

Most of the time actions like this can only be performed in Windows, is that correct?

 

[9bitbyte]

I don't see the point in adding encryption later. Atleast not for SSDs. All SSDs got some spare area that you can't access but where your SSD will store your data. So lets say every SSD is actually 10% bigger than sold as (used for wear leveling and caching), so a 1000 GB SSD actually got 1100GB of storage but where 100GB aren't accessible. If you now wipe that SSD to delete old unencrypted data or if your overwrite 1TB of unencrypted data with encrypted data, there is still up to 100GB of unencrypted data remaining in the unaccessible area which you can't overwrite but could contain your clear text passwords or other sensitive data that still would remain on your "fully" encrypted drive. If you really want to get rid of all your old unencrypted data you need to tell the SSDs firmware to do a "secure erase" where everything will be set back to factory defaults which should include deleting the complete 1100GB including this spare area that you can't overwrite otherwise. Only if you know there will be no unencrypted data left on the drive it makes sense to actually encrypt it.

I am a little bit confused! Is this a response to my question about encrypting an existing 2xssd zfs mirror rpool?