Encrypting Proxmox VE (Best Methods)

I am a little bit confused! Is this a response to my question about encrypting an existing 2xssd zfs mirror rpool?
Yes, wanted to show the problem with encrypting a SSD that stored unencrypted data before. For security reasons you should always "secure erase" the SSDs first before encrypting them so there would be nothing left on the SSDs and "encrypting an existing 2xssd zfs mirror rpool" would be useless.
 
  • Like
Reactions: 9bitbyte
Yes, wanted to show the problem with encrypting a SSD that stored unencrypted data before. For security reasons you should always "secure erase" the SSDs first before encrypting them so there would be nothing left on the SSDs and "encrypting an existing 2xssd zfs mirror rpool" would be useless.
Now I understand, thanks!

The poroxmox (re)installation didn't/wouldn't securely erase them?

Also, does this vulnerability apply only to zfs native encryption? I.e. would a LUKS full disk encryption protect my SSDs without exposing such vulnerability?
 
No, its a problem in general, no matter what encryption you use. And doing a secure erase isn't that easy. Like I already said, there is no way for the operating system to access all areas of a SSD so a OS also can't fully delete/overwrite it. You really need to tell the firmware of the SSD to do a "secure erase" operation and than the SSDs by itself will (hopefully) wipe everything using its build in algorithms. You basically need to trigger a buildin factory reset function and the SSD will wipe itself on its own. Only the SSDs firmware can access this hidden spare area, so the firmware is also the only thing which can wipe it and delete your unencrypted sensitive data stored there.
Here is shown how to do it with hdparm in linux for sata SSDs. Often the manufacturers will offer a tool with GUI to manage their SSD. But often (like Samsung Magican for Samsung SSDs) these tools will only run on Windows and there is no linux/unix support.
 
Last edited:
  • Like
Reactions: 9bitbyte
No, its a problem in general, no matter what encryption you use. And doing a secure erase isn't that easy. Like I already said, there is no way for the operating system to access all areas of a SSD so a OS also can't fully delete/overwrite it. You really need to tell the firmware of the SSD to do a "secure erase" operation and than the SSDs by itself will (hopefully) wipe everything using its build in algorithms. You basically need to trigger a buildin factory reset function and the SSD will wipe itself on its own. Only the SSDs firmware can access this hidden spare area, so the firmware is also the only thing which can wipe it and delete your unencrypted sensitive data stored there.
Here is shown how to do it with hdparm in linux for sata SSDs. Often the manufacturers will offer a tool with GUI to manage their SSD. But often (like Samsung Magican for Samsung SSDs) these tools will only run on Windows and there is no linux/unix support.
Thanks for pointing this out, it's crystal clear now! I will need to secure erase... (o_O)
 
If using LUKS for full disk encryption, is it better to encrypt the partition or the physical volume (I may have named them wrong)? i.e. is it best practice to encrypt sda sdb sdc, or sda1 sdb1 sdc1? Assuming they are the single partition on the disk and take up the full quota of usable storage space.

I read somewhere that it’s best practice to have a partition even when it’s the only one with the whole disk used, as some software may have trouble detecting and try to create one (although I have never experienced that issue). If the main sda sdb sdc are encrypted then I see no point in the partition existing as it can’t be seen without being decrypted anyway.

Someone please enlighten me :)
 
Last edited:
The structure I see everywhere is:

sda
|__sda1
……|__volume_group
…………|__logical_volume

In what way is that better than:

sda
…|__volume_group
………|__logical_volume
 
Last edited:
If you got an partition PVE won't allow you use it to create a LVM/ZFS with it so it would prevent you from accidently destroying your encrypted disk.
 
If you got an partition PVE won't allow you use it to create a LVM/ZFS with it so it would prevent you from accidently destroying your encrypted disk.
If the volume group is already setup then it’s the same either way, no?
 
Ok I have another question :)
Technically more of a Debian question but as I’m installing Proxmox over it’s a valid one…

Is it possible to install Debian onto LUKS encrypted LVM with a gpt partition table, sha512, and self-created custom partitions? The Debian installer formats as “dos” (mbr) and the encryption options only allow up to sha256. Also it wants us to use the installer to create encrypted LVM rather than install on a disk that has already been setup.
 
Did you choose the "Graphical expert install" under the "Advanced option..." entry when booting from the Debian ISO?
That will give you alot of more options.
And you can use the installer to manually partition your disk instead of using the guided partitioning.
I for example got 2 disks that I wanted to use as a mirror. Your boot partitions need to be unencrypted, So I first created a sda1 and sdb1 with each 1MB as my grub partition.
Then I created the partition sda2 and sdb2 of 300MB (you really should use 512MB nowadays) each for my ESP.
Then I created sda3 and sdb3 as a mdadm raid1 array for my unencrypted but mirrored boot partition.
Then I created sda4 and sdb4 as a mdadm raid1 array for my LUKS.
Ontop of that LUKS I created a LVM.
Ontop of that LVM my VG.
Ontop of that VG my Swap and my root LVs.

And that all can be done using the GUI of the Debian installer so this will be created:
Code:
root@Hypervisor:~# lsblk
NAME                              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                 8:0    0  93.2G  0 disk
├─sda1                              8:1    0     1M  0 part
├─sda2                              8:2    0   286M  0 part
├─sda3                              8:3    0   488M  0 part
│ └─md0                             9:0    0   487M  0 raid1 /boot
└─sda4                              8:4    0  92.4G  0 part
  └─md1                             9:1    0  92.3G  0 raid1
    └─md1_crypt                   253:0    0  92.3G  0 crypt
      ├─vgpmx-lvroot              253:1    0    21G  0 lvm   /
      └─vgpmx-lvswap              253:2    0    61G  0 lvm   [SWAP]
sdb                                 8:16   0  93.2G  0 disk
├─sdb1                              8:17   0     1M  0 part
├─sdb2                              8:18   0   286M  0 part  /boot/efi
├─sdb3                              8:19   0   488M  0 part
│ └─md0                             9:0    0   487M  0 raid1 /boot
└─sdb4                              8:20   0  92.4G  0 part
  └─md1                             9:1    0  92.3G  0 raid1
    └─md1_crypt                   253:0    0  92.3G  0 crypt
      ├─vgpmx-lvroot              253:1    0    21G  0 lvm   /
      └─vgpmx-lvswap              253:2    0    61G  0 lvm   [SWAP]
 
Last edited:
  • Like
Reactions: Whitterquick
Did you choose the "Graphical expert install" under the "Advanced option..." entry when booting from the Debian ISO?
That will give you alot of more options.
And you can use the installer to manually partition your disk instead of using the guided partitioning.
I for example got 2 disks that I wanted to use as a mirror. Your boot partitions need to be unencrypted, So I first created a sda1 and sdb1 with each 1MB as my grub partition.
Then I created the partition sda2 and sdb2 of 300MB (you really should use 512MB nowadays) each for my ESP.
Then I created sda3 and sdb3 as a mdadm raid1 array for my unencrypted but mirrored boot partition.
Then I created sda4 and sda5 as a mdadm raid1 array for my LUKS.
Ontop of that LUKS I created a LVM.
Ontop of that LVM my VG.
Optop of that VG my Swap and my root LVs.

And that all can be done using the GUI of the Debian installer so this will be created:
Code:
root@Hypervisor:~# lsblk
NAME                              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                 8:0    0  93.2G  0 disk
├─sda1                              8:1    0     1M  0 part
├─sda2                              8:2    0   286M  0 part
├─sda3                              8:3    0   488M  0 part
│ └─md0                             9:0    0   487M  0 raid1 /boot
└─sda4                              8:4    0  92.4G  0 part
  └─md1                             9:1    0  92.3G  0 raid1
    └─md1_crypt                   253:0    0  92.3G  0 crypt
      ├─vgpmx-lvroot              253:1    0    21G  0 lvm   /
      └─vgpmx-lvswap              253:2    0    61G  0 lvm   [SWAP]
sdb                                 8:16   0  93.2G  0 disk
├─sdb1                              8:17   0     1M  0 part
├─sdb2                              8:18   0   286M  0 part  /boot/efi
├─sdb3                              8:19   0   488M  0 part
│ └─md0                             9:0    0   487M  0 raid1 /boot
└─sdb4                              8:20   0  92.4G  0 part
  └─md1                             9:1    0  92.3G  0 raid1
    └─md1_crypt                   253:0    0  92.3G  0 crypt
      ├─vgpmx-lvroot              253:1    0    21G  0 lvm   /
      └─vgpmx-lvswap              253:2    0    61G  0 lvm   [SWAP]
I have not looked into mdadm or RAID yet. I also see the expert installer only allows up to sha256.

Is 1M enough for GRUB? How do you get it on there once you have partitioned? I didn’t see any option to install the boot loader on multiple partitions…
 
Is 1M enough for GRUB?
Here it was fine so far. But you can ofcause use more. Using 8MB and loosing 7MB more space probably won't hurt you ;)
How do you get it on there once you have partitioned? I didn’t see any option to install the boot loader on multiple partitions…
According to my notes I've choosen while installing...
- Install Grub to MBR: Yes
- Device for boot loader: "/dev/sda"
- Force GRUB to removable media path: yes
- Finish installation
...so the bootloader was just installed to sda. But after logging in as root I ran dpkg-reconfigure grub-pc and there I selected "/dev/sda" and "/dev/sdb" as the target so that grub gets installed on both disks.
 
  • Like
Reactions: Whitterquick
As I've a fully encrypted Ubuntu 20.04 with zfs running, I wanted to test if the same would be possible for proxmox, because I want to migrate the ubuntu installation over to proxmox.

Im not the biggest fan of the luks + zfs version, because of the added complexity, while it should be possible only with zfs. So I searched a while and found some bits and pieces her in the forums and the zfs repo.

Right now I have the root of my proxmox server encrypted and the password is asked at boot via console and ssh. So I can input the password locally or remotely.

Because I found no complete tutorial, I created a forum account to share with you what I've done so far.

Warning: You will need zfs 0.8.5 or newer, as far as I know. ,So this will only work proxmox 6.3 or newer. I've tested it with the newly released 6.4 version.
  1. Install proxmox and choose zfs raid 0 in the installer.
  2. After installation I booted right into the rescue mode and followed this link to a gist with instructions(which is originally from this forum. Thanks @Yakuraku)
    • In addition to the tutorial, I had to move the mount point of the dataset "copyroot/pve-1" to /rpool/copyroot/pve-1 to get work. Because without this, there where two datasets with the same mount point and the system didn't boot.
    • Another way would be to delete delete the old copyroot stuff after migrating. That's what you would like to do anyways.
  3. After that I was able to boot into the system.
  4. Because I want to unlock my encrypted root dataset remotely, there was one other step to do. Since zfs 0.8.5 it's possible to unlock zfs datasets via ssh while the system is booting. It's fairly simple, but a bit harder to find. Just follow the instructions at the bottom of the initramfs readme from zfs github repo
  5. After that there was only one more step to do: encrypting the dataset for vms and containers. That's well documented in the proxmox documentation
    1. Create an encryption key:
      Code:
      openssl rand -hex -out /root/data-encrypted.key 32
    2. Create new encrypted dataset:
      Code:
      zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///root/data-encrypted.key rpool/data/encrypted
    3. Add the encrypted dataset to proxmox:
      Code:
      pvesm add zfspool encrypted_zfs -pool rpool/data/encrypted

I hope that little writeup helps anyone.
Someone knows if that still works fine with the new proxmox-boot-tool? Should replacing a failed disk work the same way like described in the wiki?

Changing a failed device​

# zpool replace -f <pool> <old device> <new device>

Changing a failed bootable device​

Depending on how Proxmox VE was installed it is either using proxmox-boot-tool [1] or plain grub as bootloader (see Host Bootloader). You can check by running:
# proxmox-boot-tool status
The first steps of copying the partition table, reissuing GUIDs and replacing the ZFS partition are the same. To make the system bootable from the new disk, different steps are needed which depend on the bootloader in use.
# sgdisk <healthy bootable device> -R <new device>
# sgdisk -G <new device>
# zpool replace -f <pool> <old zfs partition> <new zfs partition>
Use the zpool status -v command to monitor how far the resilvering process of the new disk has progressed.
With proxmox-boot-tool:
# proxmox-boot-tool format <new disk's ESP>
# proxmox-boot-tool init <new disk's ESP>
ESP stands for EFI System Partition, which is setup as partition #2 on bootable disks setup by the Proxmox VE installer since version 5.4. For details, see Setting up a new partition for use as synced ESP.
With grub:
# grub-install <new disk>
 
Someone knows if that still works fine with the new proxmox-boot-tool? Should replacing a failed disk work the same way like described in the wiki?
I'm looking into booting Proxmox from an encrypted ZFS dataset (being a mirror), so I'm also interested in this question.

@Dunuin Did you investigate this and have some insights? :)
 
For anyone following this thread and wondering (like I was), I tried the recovery instructions posted above in an OVMF guest. I set up an encrypted ZFS boot mirror, using @nschemel's instructions, for the test. It worked fine; after removing a vdisk from the mirror, and adding a new vdisk, I followed the instructions for proxmox-boot-tool and was able to boot from the new vdisk.
 
  • Like
Reactions: Dunuin
I also just set up an encrypted PVE with ZFS following @nschemel's instructions. Worked absolutely fine so far with PVE 7.2. I'm also writing a detailed tutorial while doing it. Will post that here when it is ready.
 
  • Like
Reactions: j.io
I also just set up an encrypted PVE with ZFS following @nschemel's instructions. Worked absolutely fine so far with PVE 7.2. I'm also writing a detailed tutorial while doing it. Will post that here when it is ready.
Thank you! Looking forward to the tutorial.
 
But be warned...I wasn't able to keep in short and on the point...right now it's more a complete guide from creating the install USB stick, over a way too long excessive monologue about ZFS fundamentals to the complete configuration of the encrypted PVE installation including LUKS encrypted swap and remote unlocking with VLAN support.
 
  • Like
Reactions: j.io

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!