[TUTORIAL] Adding Full Disk Encryption to Proxmox

dmpm · Sunday at 18:01

esi_y said:
That's a really nice find! I completely missed it, SED support now from cryptsetup!
https://lore.kernel.org/all/cd409f6c-5d51-482c-8a26-340822754ff1@gmail.com/T/

I am literally curious how that will work out for you now.

I used Xubuntu minimal in live mode, which already had dosfstools cryptsetup (v2.7) and efibootmgr installed so I didn't have to install those.

On my server, when I install PVE it creates the 1M BIOS boot partition starting at sector 34, but on my PC where I'm doing this (my server is a Lenovo M710 Tiny which only has one SSD SATA port) fdisk only allows me to start creating partitions at sector 2048, so I skipped the BIOS boot partition as you suggest and just created three partitions.

However, when I tried to encrypt the third partition with 'cryptsetup luksFormat --hw-opal-only /dev/sda3' I got an error "Cannot setup OPAL segment"

so I download and untarred sedutil-cli and ran 'sedutil-cli --scan' and that identified that "The Kernel flag libata.allow_tpm is not set correctly",

That needs to be set in the kernel parameters or by editing it on the running system.
https://wiki.archlinux.org/title/Self-encrypting_drives#Using_sedutil

The way to edit it on the running system is:

cd /sys/module/libata/parameters
sudo chmod 644 allow_tpm
sudo nano allow_tpm
Delete number 0 and write number 1. Save file.

but even though I'm logged in as root on Xubuntu live, when I tried to sudo nano allow_tpm it says "Error writing lock file ./.allow_tpm.swp: Permission denied ]" so I can't edit and save it. I can't permanently change the kernel parameters when booting Xubuntu live either, and it was getting a bit tedious having to keep rebooting and reinstalling stuff at this point, so I installed a fresh copy of PVE on a spare SSD on my server, and booted my PC from that, and then edited /etc/default/grub to add the libata.allow_tpm=1 parameter and regenerated grub.cfg with 'grub-mkconfig -o /boot/grub/grub.cfg' and rebooted.

After installing sedutil-cli I ran 'sedutil-cli --scan' to confirm that it's no longer warning about the flag not being set correctly. However, when I try to run cryptsetup it says its not installed, and when I try to install it with apt install it says:

"Package cryptsetup is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'cryptsetup' has no installation candidate"

which is weird, because it was definitely installed on my previous PVE. That was v8.2.7 though and this new install is v8.2.2. I've installed ttecks's PVE Post Install script from here https://tteck.github.io/Proxmox/#proxmox-ve-tools and it's added the no-subscription repository, but apt update, apt upgrade, apt dist-upgrade all say it's up to date, so I'll have to work out how to update it to v8.2.7 and then see if cryptsetup is installed after that.

EDIT: It's showing that it's v8.2.7 now and I can install cryptsetup with apt install, but it's v2.6 which doesn't have the OPAL support, so I still don't know how I had v2.7 on my previous PVE install.

EDIT2: OK, I discovered how I'd done it. I'd edited /etc/apt/sources.list and added 'deb http://ftp.de.debian.org/debian sid main' to the end. Adding that and running 'apt update' then allows me to upgrade cryptsetup to v2.7.5 with 'apt install cryptsetup'.

dmpm · Sunday at 18:41

I'm still not having any luck.

'cryptsetup luksFormat --hw-opal-only /dev/sda3' now returns "Blkid scan failed for /dev/sda3"

and indeed blkid doesn't show /dev/sda3 (sdb is my PVE boot drive)

/dev/mapper/pve-root: UUID="0077729e-6bb7-4c8d-8a1f-756f8476b611" BLOCK_SIZE="4096" TYPE="ext4"
/dev/nvme0n1p1: LABEL="MP600 p1" BLOCK_SIZE="512" UUID="01D818B1BA833A70" TYPE="ntfs" PARTUUID="74cc9561-01"
/dev/sdb2: UUID="26C8-7E9F" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="bf35e978-8c69-4546-ba41-ac80e30e5849"

/dev/sdb3: UUID="MNCLqh-InX5-rgaW-IM5l-8UBS-tu4Z-Zj9lJM" TYPE="LVM2_member" PARTUUID="66030e28-d9a3-414d-a0a3-8ab9806d9ce5"

/dev/mapper/pve-swap: UUID="93b54178-7bb6-4571-8bbc-1d7a02de9f98" TYPE="swap"
/dev/nvme1n1p1: LABEL="WD SN850X" BLOCK_SIZE="512" UUID="5C1AA9A11AA97922" TYPE="ntfs" PARTUUID="cb2b6ac1-01"

/dev/sda2: UUID="20a81185-0f5a-4e33-91d6-d6e9421e192b" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="23b584c1-3393-43bf-8265-16424d7dcaaa"

/dev/sda1: UUID="D520-2ADB" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="9ae0d8b7-0aad-4a8c-a9b8-c1fe0e2ad17b"
/dev/nvme0n1p2: PARTUUID="74cc9561-02"
/dev/sdb1: PARTUUID="ce2f3a6e-1673-4b82-95c5-74bce09be058"

but 'fdisk /dev/sda -l' shows /dev/sda3

Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: CT1000MX500SSD1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 20F80DAD-C185-4B0D-A46D-7EC6B0F895BD

Device Start End Sectors Size Type
/dev/sda1 2048 411647 409600 200M EFI System
/dev/sda2 411648 2050047 1638400 800M Linux filesystem
/dev/sda3 2050048 1742686207 1740636160 830G Linux filesystem

dmpm · Sunday at 19:52

Booting back to Xubuntu live, blkid doesn't show /dev/sda3 there either but 'sudo blkid' does, and I can do 'sudo cryptsetup luksFormat --hw-opal-only /dev/sda3" and that works, but when I try 'sudo cryptsetup open /dev/sda3 sda3_crypt' it says it's not a valid LUKS device.

If I format it without --hw-opal-only I can open it OK. Maybe this is a bug in cryptsetup v2.7 though and it's been fixed in v2.7.5.

In PVE I'm already running as root, so I can't do 'sudo blkid' and /dev/sda3 should be shown when I run blkid, but it isn't so I can't format or open it with cryptsetup.

esi_y · Monday at 05:18

dmpm said:
EDIT2: OK, I discovered how I'd done it. I'd edited /etc/apt/sources.list and added 'deb http://ftp.de.debian.org/debian sid main' to the end. Adding that and running 'apt update' then allows me to upgrade cryptsetup to v2.7.5 with 'apt install cryptsetup'.

That's a pretty bold way of doing it, you could just get it from e.g. trixie:
https://packages.debian.org/trixie/amd64/cryptsetup/download

dmpm said:
'cryptsetup luksFormat --hw-opal-only /dev/sda3' now returns "Blkid scan failed for /dev/sda3"

and indeed blkid doesn't show /dev/sda3 (sdb is my PVE boot drive)

but 'fdisk /dev/sda -l' shows /dev/sda3

Try partprobe: https://manpages.debian.org/buster/parted/partprobe.8.en.html

BTW Have to go now, but that was quite a nice piece of work on your part! Keep it up!!

dmpm · Monday at 11:35

esi_y said:
That's a pretty bold way of doing it, you could just get it from e.g. trixie:
https://packages.debian.org/trixie/amd64/cryptsetup/download

Yeah, I must have read a suggestion to add the sid repo somewhere. It might not have even been primarily for cryptsetup, I can't recall. I guess it doesn't really matter if I disable it again after installing cryptsetup, but just to be sure I haven't got a dodgy experimental version installed, I removed cryptsetup, adding the trixie repo instead of sid, and reinstalled cryptsetup.

I note when running 'cryptsetup --version' it returns "cryptsetup 2.7.5 flags: UDEV BLKID KEYRING KERNEL_CAPI HW_OPAL". I guess those flags indicate optional features which are enabled, including HW_OPAL, so that should work if only it could see /dev/sda3.

esi_y said:
Try partprobe: https://manpages.debian.org/buster/parted/partprobe.8.en.html

BTW Have to go now, but that was quite a nice piece of work on your part! Keep it up!!

'partprobe -s' returns

/dev/sda: gpt partitions 1 2 3
/dev/sdb: gpt partitions 1 2 3
/dev/mapper/pve-swap: loop partitions 1
/dev/mapper/pve-root: loop partitions 1
/dev/mapper/pve-data_tmeta: msdos partitions
/dev/mapper/pve-data_tdata: msdos partitions
/dev/mapper/pve-data: msdos partitions
/dev/nvme0n1: msdos partitions 1 2
/dev/nvme1n1: msdos partitions 1

so that recognises all three partitions on /dev/sda, as does lsblk, it's just blkid that doesn't.

'ls /dev/disk/by-partuuid -l' returns:

total 0
lrwxrwxrwx 1 root root 10 Oct 14 10:16 23b584c1-3393-43bf-8265-16424d7dcaaa -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 14 10:01 66030e28-d9a3-414d-a0a3-8ab9806d9ce5 -> ../../sdb3
lrwxrwxrwx 1 root root 15 Oct 14 10:16 74cc9561-01 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 14 10:16 74cc9561-02 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 10 Oct 14 10:16 9ae0d8b7-0aad-4a8c-a9b8-c1fe0e2ad17b -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 14 10:01 bf35e978-8c69-4546-ba41-ac80e30e5849 -> ../../sdb2
lrwxrwxrwx 1 root root 15 Oct 14 10:16 cb2b6ac1-01 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 10 Oct 14 10:01 ce2f3a6e-1673-4b82-95c5-74bce09be058 -> ../../sdb1

and 'ls /dev/disk/by-uuid -l' returns:

total 0
lrwxrwxrwx 1 root root 10 Oct 14 10:01 0077729e-6bb7-4c8d-8a1f-756f8476b611 -> ../../dm-1
lrwxrwxrwx 1 root root 15 Oct 14 10:16 01D818B1BA833A70 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 10 Oct 14 10:16 20a81185-0f5a-4e33-91d6-d6e9421e192b -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 14 10:01 26C8-7E9F -> ../../sdb2
lrwxrwxrwx 1 root root 15 Oct 14 10:16 5C1AA9A11AA97922 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 10 Oct 14 10:01 93b54178-7bb6-4571-8bbc-1d7a02de9f98 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Oct 14 10:16 D520-2ADB -> ../../sda1

so neither show /sda3, but ' ls /dev/disk/by-id -l' returns:

total 0
lrwxrwxrwx 1 root root 9 Oct 14 10:01 ata-Crucial_CT256MX100SSD1_14270C8AB149 -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct 14 10:01 ata-Crucial_CT256MX100SSD1_14270C8AB149-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Oct 14 10:01 ata-Crucial_CT256MX100SSD1_14270C8AB149-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Oct 14 10:01 ata-Crucial_CT256MX100SSD1_14270C8AB149-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Oct 14 10:01 ata-CT1000MX500SSD1_2422E8B69443 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct 14 10:16 ata-CT1000MX500SSD1_2422E8B69443-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 14 10:16 ata-CT1000MX500SSD1_2422E8B69443-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 14 10:16 ata-CT1000MX500SSD1_2422E8B69443-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Oct 14 10:01 dm-name-pve-root -> ../../dm-1
lrwxrwxrwx 1 root root 10 Oct 14 10:01 dm-name-pve-swap -> ../../dm-0

lrwxrwxrwx 1 root root 10 Oct 14 10:01 dm-uuid-LVM-AG3L1ByDNXSwmAlmfWrHwit9JHQ7LOI80WtN4b4jimflJFiVN2G4qkaLZ6s8gul2 -> ../../dm-0

lrwxrwxrwx 1 root root 10 Oct 14 10:01 dm-uuid-LVM-AG3L1ByDNXSwmAlmfWrHwit9JHQ7LOI87Ox44hBRqcZSZH82XQs4ro3mmLoi3KLj -> ../../dm-1

lrwxrwxrwx 1 root root 10 Oct 14 10:01 lvm-pv-uuid-MNCLqh-InX5-rgaW-IM5l-8UBS-tu4Z-Zj9lJM -> ../../sdb3
lrwxrwxrwx 1 root root 13 Oct 14 10:01 nvme-eui.e8238fa6bf530001001b444a48bf9c06 -> ../../nvme1n1
lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-eui.e8238fa6bf530001001b444a48bf9c06-part1 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 13 Oct 14 10:01 nvme-Force_MP600_20378230000128564636 -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Oct 14 10:01 nvme-Force_MP600_20378230000128564636_1 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-Force_MP600_20378230000128564636_1-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-Force_MP600_20378230000128564636_1-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-Force_MP600_20378230000128564636-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-Force_MP600_20378230000128564636-part2 -> ../../nvme0n1p2

lrwxrwxrwx 1 root root 13 Oct 14 10:01 nvme-nvme.1987-3230333738323330303030313238353634363336-466f726365204d50363030-00000001 -> ../../nvme0n1

lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-nvme.1987-3230333738323330303030313238353634363336-466f726365204d50363030-00000001-part1 -> ../../nvme0n1p1

lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-nvme.1987-3230333738323330303030313238353634363336-466f726365204d50363030-00000001-part2 -> ../../nvme0n1p2

lrwxrwxrwx 1 root root 13 Oct 14 10:01 nvme-WD_BLACK_SN850X_2000GB_225189443714 -> ../../nvme1n1
lrwxrwxrwx 1 root root 13 Oct 14 10:01 nvme-WD_BLACK_SN850X_2000GB_225189443714_1 -> ../../nvme1n1
lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-WD_BLACK_SN850X_2000GB_225189443714_1-part1 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 15 Oct 14 10:16 nvme-WD_BLACK_SN850X_2000GB_225189443714-part1 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 9 Oct 14 10:01 wwn-0x500a07510c8ab149 -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct 14 10:01 wwn-0x500a07510c8ab149-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Oct 14 10:01 wwn-0x500a07510c8ab149-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Oct 14 10:01 wwn-0x500a07510c8ab149-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Oct 14 10:01 wwn-0x500a0751e8b69443 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct 14 10:16 wwn-0x500a0751e8b69443-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 14 10:16 wwn-0x500a0751e8b69443-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 14 10:16 wwn-0x500a0751e8b69443-part3 -> ../../sda3

so it shows two entries for /sda3, one under ata-CT1000MX500SSD1_2422E8B69443 and the other under wwn-0x500a0751e8b69443. However, every drive and partition has at least two entries, so that seems to be normal.

EDIT: This page explains that partitions only have a UUID if they're formatted, but it seems a bit strange to have to format a partition as ext4 just to be able to encrypt it with 'cryptsetup luksFormat', after which you might want to format it as ZFS or something.
https://unix.stackexchange.com/ques...hat-wont-show-up-in-dev-disk-by-uuid-or-blkid

I tried running 'mkfs -t ext4 /dev/sda3' and it took ages before failing anyway ( the Creating journal step took a long while, and the Writing superblocks step got stuck on 0/x for ages until it failed):

mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: failed - Input/output error
Creating filesystem with 217579520 4k blocks and 54394880 inodes
Filesystem UUID: a6b5be33-6967-4287-9045-4244747b88c3
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done

Writing superblocks and filesystem accounting information: mkfs.ext4: Input/output error while writing out and closing file system

dmpm · Monday at 23:28

I've made some progress. I connected the new SSD to my M710 Tiny server and booted Ubunutu live on that (after editing grub.cfg to add the libata.allow_tpm=1 flag) and then added the trixie repo to /etc/apt/sources.list (with the [trusted=yes] flag) and updated cryptsetup to 2.7.5.1.

I was still getting the "Can't set OPAL segment" error when running 'cryptsetup luksFormat --hw-opal-only /dev/sda3' so I tried deleting all the partitions and just creating one to see if that helped, but it didn't. So I did a PSID erase with 'cryptsetup erase --hw-opal-factory-reset --key-file PSID.txt /dev/sda' (it's a lot easier to type the PSID in a file and use that rather than trying to type it blind each time, because it doesn't display when you do that).

Strangely lsblk still showed the partition after that, but fdisk showed that there were none so I guess lsblk just doesn't update properly. Anyway, I was able to create the three partitions and format the third one with 'cryptsetup luksFormat --hw-opal-only /dev/sda3' after that. I'm not sure why I had to do another PSID reset, as I already did that on the PC to reset it after I used sedutil to lock it originally, before realising that wouldn't work for my situation and I needed to use cryptsetup instead to be able to auto/remote unlock at boot.

I'm stuck on the copying old installation across stage now though. Because my M710 server only has one SATA port, I've put the old drive (with a clean install of PVE on it) in a USB caddy and connected that up, and that's working and lsblk now shows:

sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 200M 0 part
├─sda2 8:2 0 800M 0 part
└─sda3 8:3 0 830G 0 part
└─cryptlvm 252:0 0 830G 0 crypt
├─pve--new-root 252:1 0 30G 0 lvm /mnt/new
└─pve--new-swap 252:2 0 8G 0 lvm
sdb 8:16 1 28.7G 0 disk
└─sdb1 8:17 1 28.7G 0 part /cdrom
sde 8:64 0 238.5G 0 disk
├─sde1 8:65 0 1007K 0 part
├─sde2 8:66 0 1G 0 part
└─sde3 8:67 0 237.5G 0 part
├─pve-data_tmeta 252:5 0 1.4G 0 lvm
│ └─pve-data 252:7 0 141.2G 0 lvm
└─pve-data_tdata 252:6 0 141.2G 0 lvm
└─pve-data 252:7 0 141.2G 0 lvm

Most of the mount commands work after using the correct partitions, so for example for the last command I used 'mount /dev/sde2 /mnt/old/boot/efi', but the one before it, 'mount /dev/pve/root /mnt/old' doesn't work and returns:

mount: /mnt/old: can't read superblock on /dev/mapper/pve-root.
dmesg(1) may have more information after failed mount system call.

lvs shows that pve/root exists, so I don't know why this isn't working.

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-a-tz-- 141.22g 0.00 1.13
root pve -wi-a----- <69.37g
swap pve -wi-a----- 8.00g
root pve-new -wi-ao---- 30.00g
swap pve-new -wi-a----- 8.00g

dmesg shows (this was after the USB caddy was disconnected and reconnected, so the drive letter changed to /dev/sdf)

buffer_io_error: 8 callbacks suppressed
[ 7722.079686] Buffer I/O error on dev dm-4, logical block 0, async page read
[ 7722.080126] EXT4-fs (dm-4): unable to read superblock
[ 7737.866724] /dev/sdf2: Can't lookup blockdev
[ 7755.954886] FAT-fs (sdd2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 7831.929415] Buffer I/O error on dev dm-4, logical block 0, async page read
[ 7831.929905] EXT4-fs (dm-4): unable to read superblock

I'll try connecting both drives back to my PC via SATA now that I've got the partition encrypted, and see if it works on there.

esi_y · Monday at 23:35

I can see your updates here, just not enough time to follow instantly. Keep it up!

BTW I mostly thought you would only get that one package from trixie, download it and dpkg -i. I thought I would have closer look at those flags and what that (might) also need from kernel.

The mismatch what e.g. lsblk shows you should be literally a partprobe away, but I have to look at why it failed for you above.

dmpm · 2024-10-15T00:32:47+0200

esi_y said:
I can see your updates here, just not enough time to follow instantly. Keep it up!

BTW I mostly thought you would only get that one package from trixie, download it and dpkg -i. I thought I would have closer look at those flags and what that (might) also need from kernel.

The mismatch what e.g. lsblk shows you should be literally a partprobe away, but I have to look at why it failed for you above.

I did try downloading the deb after you pointed me to it (I searched several times before that but could only find packages for other distros and nothing for Debian, so obviously my Google-Fu isn't very good!) but it failed to install due to missing dependencies, and using apt with the trixie repo added sorts all that out.

I think the fact that /dev/sda3 isn't showing in blkid under Proxmox is a red herring, as it doesn't show when I try in Ubuntu either but I can still format the partition with cryptsetup, so it seems like it's something else in Proxmox was preventing it working. The kernel with Ubuntu live is 6.8.0-41-generic and I can't recall what the Proxmox one is, but I think it was recent enough to support cryptsetup 2.7.5.1 and the OPAL features.

EDIT: OK, after connecting the drives to my PC via SATA and booting Ubuntu live it worked. I was able to open the encrypted partition with cryptsetup v2.7.0 and didn't need to update it to v.2.7.5.1 and the old pve-root mounted fine, so that problem must have been caused by the USB caddy.

This is what df -h shows after copying the contents across:

Filesystem Size Used Avail Use% Mounted on
tmpfs 6.3G 2.6M 6.3G 1% /run
efivarfs 128K 51K 73K 42% /sys/firmware/efi/efivars
/dev/sdc1 29G 5.8G 23G 21% /cdrom
/cow 32G 521M 31G 2% /
tmpfs 32G 8.0K 32G 1% /dev/shm
tmpfs 5.0M 12K 5.0M 1% /run/lock
tmpfs 32G 0 32G 0% /tmp
tmpfs 6.3G 160K 6.3G 1% /run/user/1000
tmpfs 6.3G 84K 6.3G 1% /run/user/0
/dev/mapper/pve--new-root 30G 3.7G 25G 14% /mnt/new
/dev/sdb2 770M 176M 538M 25% /mnt/new/boot
/dev/sdb1 197M 12M 186M 6% /mnt/new/boot/efi
/dev/mapper/pve-root 68G 3.9G 61G 6% /mnt/old
/dev/sda2 1022M 12M 1011M 2% /mnt/old/boot/efi

The rest went smoothly. The only thing that came up was when I ran 'update-initramfs -c -k all' and it returned:

update-initramfs: Generating /boot/initrd.img-6.8.12-2-pve
cryptsetup: WARNING: Couldn't determine cipher modules to load for cryptlvm
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-6.8.4-2-pve
cryptsetup: WARNING: Couldn't determine cipher modules to load for cryptlvm
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

so I'll have to see if the cryptsetup warning causes any problems.

Is there a typo under the "SSH Unlock" section? It says 'echo 'ssh-rsa ...' >> /etc/dropbear-initramfs/authorized_keys' but I only have a /etc/dropbear/initramfs/ folder and no /etc/dropbear-initramfs/ one. I skipped this step for now anyway, as I want to test that it boots first and I can sort out dropbear later.

EDIT2: I didn't really understand the efibootmgr instructions. The first time I ran 'efibootmgr -v' it returned

BootCurrent: 0004
Timeout: 1 seconds
BootOrder: 0000,0003,0004
Boot0000* proxmox HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\proxmox\shimx64.efi)

      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 40 06 00 00 00 00 00 f9 af 90 db c2 80 9c 45 8c 65 9b 4c 81 91 45 0b 02 02 / 04 04 36 00 5c 00 45 00 46 00 49 00 5c 00 70 00 72 00 6f 00 78 00 6d 00 6f 00 78 00 5c 00 73 00 68 00 69 00 6d 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00

Boot0003* UEFI OS       HD(2,GPT,bf35e978-8c69-4546-ba41-ac80e30e5849,0x800,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)0000424f

      dp: 04 01 2a 00 02 00 00 00 00 08 00 00 00 00 00 00 00 00 20 00 00 00 00 00 78 e9 35 bf 69 8c 46 45 ba 41 ac 80 e3 0e 58 49 02 02 / 04 04 30 00 5c 00 45 00 46 00 49 00 5c 00 42 00 4f 00 4f 00 54 00 5c 00 42 00 4f 00 4f 00 54 00 58 00 36 00 34 00 2e 00 45 00 46 00 49 00 00 00 / 7f ff 04 00

data: 00 00 42 4f

Boot0004* UEFI:  USB, Partition 1       PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/Pci(0x8,0x0)/Pci(0x0,0x3)/USB(6,0)/HD(1,MBR,0x15737896,0x800,0x39567e0)0000424f

      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 02 01 / 01 01 06 00 00 00 / 01 01 06 00 00 08 / 01 01 06 00 03 00 / 03 05 06 00 06 00 / 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 e0 67 95 03 00 00 00 00 96 78 73 15 00 00 00 00 00 00 00 00 00 00 00 00 01 01 / 7f ff 04 00

data: 00 00 42 4f

sda is the old drive, so I ran 'efibootmgr --create-only --disk /dev/sda --part 2 --loader /EFI/proxmox/grubx64.efi --label "Proxmox unencrypted" -v' and got:

BootCurrent: 0004
Timeout: 1 seconds
BootOrder: 0000,0003,0004
Boot0000* proxmox HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\proxmox\shimx64.efi)

      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 40 06 00 00 00 00 00 f9 af 90 db c2 80 9c 45 8c 65 9b 4c 81 91 45 0b 02 02 / 04 04 36 00 5c 00 45 00 46 00 49 00 5c 00 70 00 72 00 6f 00 78 00 6d 00 6f 00 78 00 5c 00 73 00 68 00 69 00 6d 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00

Boot0003* UEFI OS       HD(2,GPT,bf35e978-8c69-4546-ba41-ac80e30e5849,0x800,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)0000424f

      dp: 04 01 2a 00 02 00 00 00 00 08 00 00 00 00 00 00 00 00 20 00 00 00 00 00 78 e9 35 bf 69 8c 46 45 ba 41 ac 80 e3 0e 58 49 02 02 / 04 04 30 00 5c 00 45 00 46 00 49 00 5c 00 42 00 4f 00 4f 00 54 00 5c 00 42 00 4f 00 4f 00 54 00 58 00 36 00 34 00 2e 00 45 00 46 00 49 00 00 00 / 7f ff 04 00

data: 00 00 42 4f

Boot0004* UEFI:  USB, Partition 1       PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/Pci(0x8,0x0)/Pci(0x0,0x3)/USB(6,0)/HD(1,MBR,0x15737896,0x800,0x39567e0)0000424f

      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 02 01 / 01 01 06 00 00 00 / 01 01 06 00 00 08 / 01 01 06 00 03 00 / 03 05 06 00 06 00 / 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 e0 67 95 03 00 00 00 00 96 78 73 15 00 00 00 00 00 00 00 00 00 00 00 00 01 01 / 7f ff 04 00

data: 00 00 42 4f

Boot0001* Proxmox unencrypted   HD(2,GPT,bf35e978-8c69-4546-ba41-ac80e30e5849,0x800,0x200000)/File(\EFI\proxmox\grubx64.efi)

      dp: 04 01 2a 00 02 00 00 00 00 08 00 00 00 00 00 00 00 00 20 00 00 00 00 00 78 e9 35 bf 69 8c 46 45 ba 41 ac 80 e3 0e 58 49 02 02 / 04 04 36 00 5c 00 45 00 46 00 49 00 5c 00 70 00 72 00 6f 00 78 00 6d 00 6f 00 78 00 5c 00 67 00 72 00 75 00 62 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00

so the last entry was the one I added, and the bf35... UUID matched Boot0003, so I did 'efibootmgr -B -b 000X -v' and got:

BootCurrent: 0004
Timeout: 1 seconds
BootOrder: 0000,0004
Boot0000* proxmox HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\proxmox\shimx64.efi)

      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 40 06 00 00 00 00 00 f9 af 90 db c2 80 9c 45 8c 65 9b 4c 81 91 45 0b 02 02 / 04 04 36 00 5c 00 45 00 46 00 49 00 5c 00 70 00 72 00 6f 00 78 00 6d 00 6f 00 78 00 5c 00 73 00 68 00 69 00 6d 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00

Boot0001* Proxmox unencrypted   HD(2,GPT,bf35e978-8c69-4546-ba41-ac80e30e5849,0x800,0x200000)/File(\EFI\proxmox\grubx64.efi)

      dp: 04 01 2a 00 02 00 00 00 00 08 00 00 00 00 00 00 00 00 20 00 00 00 00 00 78 e9 35 bf 69 8c 46 45 ba 41 ac 80 e3 0e 58 49 02 02 / 04 04 36 00 5c 00 45 00 46 00 49 00 5c 00 70 00 72 00 6f 00 78 00 6d 00 6f 00 78 00 5c 00 67 00 72 00 75 00 62 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00

Boot0004* UEFI:  USB, Partition 1       PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/Pci(0x8,0x0)/Pci(0x0,0x3)/USB(6,0)/HD(1,MBR,0x15737896,0x800,0x39567e0)0000424f

      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 02 01 / 01 01 06 00 00 00 / 01 01 06 00 00 08 / 01 01 06 00 03 00 / 03 05 06 00 06 00 / 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 e0 67 95 03 00 00 00 00 96 78 73 15 00 00 00 00 00 00 00 00 00 00 00 00 01 01 / 7f ff 04 00

data: 00 00 42 4f

so the "Proxmox unencrypted" entry still points to the old drive. Actually, I just realised as I was typing this that I was meant to run efibootmgr on the server that I'm going to be running Proxmox on, not the PC I'm using to copy the data from the old to the new drive, but I guess it won't have done any harm.

dmpm · 2024-10-15T03:30:17+0200

When I first tried to boot, it showed the correct IP address that I set of 192.168.0.198 and the broadcast address but the gateway and DNS addresses were 0, and then it said "cryptsetup: Waiting for encrypted source device UUID=8cbbf80f....." repeatedly and failed with "Gave up waiting for root file system device" and "ALERT! /dev/mapper/pve-root does not exist"

I rebooted into Ubuntu live on the server to see if I could find the problem. blkid showed that my encrypted partition /dev/sda3 does have the UUID starting with 8cbbf80f, so that wasn't the problem. I did find that /etc/network/interfaces had the wrong IP address, gateway and network interface name, so I fixed them but that shouldn't have made any difference because that doesn't get loaded until later. Anyway, when I rebooted it loaded up a lot quicker and showed "Please unlock disk cryptlvm" at the top, mixed in with the IP-Config details, and typing in my encryption password worked and it booted proxmox fine, so that's working.

I just need to get dropbear working now. Using MobaXterm I can connect and log in as root (so the fact the gateway is showing as 0.0.0.0 clearly doesn't matter) and it shows the "Please unlock disk cryptlvm" prompt, but when I enter the password, after a while it just says "Error: Timeout reached while waiting for PID xxx".

esi_y · 2024-10-15T05:29:10+0200

dmpm said:
Is there a typo under the "SSH Unlock" section? It says 'echo 'ssh-rsa ...' >> /etc/dropbear-initramfs/authorized_keys' but I only have a /etc/dropbear/initramfs/ folder and no /etc/dropbear-initramfs/ one. I skipped this step for now anyway, as I want to test that it boots first and I can sort out dropbear later.

I do not use dropbear (more tang/clevis person), but I do remember that at some point the new default became /etc/dropbear/initramfs/

You can check what got in:

lsinitramfs /boot/initrd.img...

dmpm · 2024-10-15T16:39:54+0200

And it's stopped working again!

It's gone back to taking ages after saying it's "loading initial ramdisk" and then the IP Config info shows but the cryptsetup password prompt doesn't appear and it does the ""cryptsetup: Waiting for encrypted source device UUID=8cbbf80f....." repeatedly and fails with "Gave up waiting for root file system device" and "ALERT! /dev/mapper/pve-root does not exist".

In the BIOS, it showed two options under the drive, proxmox and UEFI OS, so I tried swapping the order but that made no difference. I tried disabling secure boot too but I still have the same cryptsetup errors. I can still login as root using dropbear, but there's no cryptsetup prompt there anymore either.

esi_y · 2024-10-15T18:04:09+0200

dmpm said:
And it's stopped working again!

It's gone back to taking ages after saying it's "loading initial ramdisk" and then the IP Config info shows but the cryptsetup password prompt doesn't appear and it does the ""cryptsetup: Waiting for encrypted source device UUID=8cbbf80f....." repeatedly and fails with "Gave up waiting for root file system device" and "ALERT! /dev/mapper/pve-root does not exist".

In the BIOS, it showed two options under the drive, proxmox and UEFI OS, so I tried swapping the order but that made no difference. I tried disabling secure boot too but I still have the same cryptsetup errors. I can still login as root using dropbear, but there's no cryptsetup prompt there anymore either.

Just to clarify, this all is only a problem with current Proxmox VE version based on Bookworm with current kernel? Were you able to have this running full-chain with e.g. Trixie or Sid alone?

dmpm · 2024-10-15T19:21:31+0200

esi_y said:
Just to clarify, this all is only a problem with current Proxmox VE version based on Bookworm with current kernel? Were you able to have this running full-chain with e.g. Trixie or Sid alone?

I don't think I've ever updated the kernel in Proxmox. I only added the Trixie or Sid repo in Ubuntu live so I could update cryptsetup, and I found I didn't even need to do that on my last run.

My current install starts in post #46, and I used my desktop PC to do a full PSID erase of the new drive and then copied the contents of the old drive (which just contained a fresh install of Proxmox) to the new one by following your guide, and after I moved the drive back to the server it booted a few times yesterday with cryptsetup working (but not dropbear), but today cryptsetup isn't asking for the password and can't find the encrypted root partition, so I can't boot it.

dmpm · 2024-10-15T23:30:54+0200

I booted into Ubuntu live again and unlocked /dev/sda3 and chrooted into the Proxmox install to see if I could find the problem, but nothing obvious stood out.

I ran 'update-initramfs -c -k all' again anyway, and the only thing that might be an issue that came up was "cryptsetup: WARNING: Couldn't determine cipher modules to load for cryptlvm", but that might just mean that it loaded all the modules because it didn't know which ones were needed.

The first entry in grub.cfg is:

menuentry 'Proxmox VE GNU/Linux' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-/dev/mapper/pve-root' {

load_video
insmod gzio
if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
insmod part_gpt
insmod ext2
set root='hd0,gpt2'
if [ x$feature_platform_search_hint = xy ]; then

          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  4c8d35cd-3cbd-4a37-8b0f-67060928dcdd

else
search --no-floppy --fs-uuid --set=root 4c8d35cd-3cbd-4a37-8b0f-67060928dcdd
fi
echo 'Loading Linux 6.8.12-2-pve ...'

        linux   /vmlinuz-6.8.12-2-pve root=/dev/mapper/pve-root ro       quiet libata.allow_tpm=1 ip=192.168.0.198:::::eno1:none

echo 'Loading initial ramdisk ...'
initrd /initrd.img-6.8.12-2-pve

I note that the guide says "You should see that the first menu entry does not have insmod lvm and instead references a partition directly". This entry refers to 'hd0,gpt2' but I'm not sure if that's correct when root is on the decrypted /dev/sda3/ as cryptlvm/pve-root.

4c8d35cd-3cbd-4a37-8b0f-67060928dcdd is the UUID for /dev/sda2, which is the boot partition.

efibootmgr -v shows:

BootOrder: 0002,0000,0001
Boot0000* proxmox HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\PROXMOX\SHIMX64.EFI)

      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 40 06 00 00 00 00 00 f9 af 90 db c2 80 9c 45 8c 65 9b 4c 81 91 45 0b 02 02 / 04 04 36 00 5c 00 45 00 46 00 49 00 5c 00 50 00 52 00 4f 00 58 00 4d 00 4f 00 58 00 5c 00 53 00 48 00 49 00 4d 00 58 00 36 00 34 00 2e 00 45 00 46 00 49 00 00 00 / 7f ff 04 00

Boot0001* UEFI OS HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\BOOT\BOOTX64.EFI)0000424f

      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 40 06 00 00 00 00 00 f9 af 90 db c2 80 9c 45 8c 65 9b 4c 81 91 45 0b 02 02 / 04 04 30 00 5c 00 45 00 46 00 49 00 5c 00 42 00 4f 00 4f 00 54 00 5c 00 42 00 4f 00 4f 00 54 00 58 00 36 00 34 00 2e 00 45 00 46 00 49 00 00 00 / 7f ff 04 00

data: 00 00 42 4f

Boot0002* UEFI:  USB, Partition 1       PciRoot(0x0)/Pci(0x14,0x0)/USB(16,0)/HD(1,MBR,0x15737896,0x800,0x39567e0)0000424f

      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 00 14 / 03 05 06 00 10 00 / 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 e0 67 95 03 00 00 00 00 96 78 73 15 00 00 00 00 00 00 00 00 00 00 00 00 01 01 / 7f ff 04 00

data: 00 00 42 4f

which I assumed was correct as I'm booted from the Ubuntu USB, so /dev/sda is HD1 rather than HD0, which it will be when I boot from the SSD.

db90aff9-80c2-459c-8c65-9b4c8191450b is the PARTUUID for the EFI partition on /dev/sda1

After removing the USB and rebooting, it's working again and prompts for the cryptlvm password and accepts it if I type it in directly. However, dropbear still gives an error after I type in the password 'Error: Timeout reached while waiting for PID 227' but I can still type it in on the server after that and it boots.

Running efibootmgr -v in proxmox shows:

BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,0001
Boot0000* proxmox HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\PROXMOX\SHIMX64.EFI)
Boot0001* UEFI OS HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

So I guess /dev/sda is HD1 as far as efibootmgr is concerned, but HD0 in grub.

esi_y · 2024-10-15T23:50:14+0200

dmpm said:
only thing that might be an issue that came up was "cryptsetup: WARNING: Couldn't determine cipher modules to load for cryptlvm", but that might just mean that it loaded all the modules because it didn't know which ones were needed.

I think that's benign, it can't read the format kernel provides it in last time i checked.

dmpm said:

Code:

menuentry 'Proxmox VE GNU/Linux' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-/dev/mapper/pve-root' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  4c8d35cd-3cbd-4a37-8b0f-67060928dcdd
        else
          search --no-floppy --fs-uuid --set=root 4c8d35cd-3cbd-4a37-8b0f-67060928dcdd
        fi
        echo    'Loading Linux 6.8.12-2-pve ...'
        linux   /vmlinuz-6.8.12-2-pve root=/dev/mapper/pve-root ro       quiet libata.allow_tpm=1 ip=192.168.0.198:::::eno1:none
        echo    'Loading initial ramdisk ...'
        initrd  /initrd.img-6.8.12-2-pve

I note that the guide says "You should see that the first menu entry does not have insmod lvm and instead references a partition directly". This entry refers to 'hd0,gpt2' but I'm not sure if that's correct when root is on the decrypted /dev/sda3/ as cryptlvm/pve-root.

I am not familiar with the guide to the dot, but I am pretty sure this was just emphasizing that you have LUKS on GPT partition in which you then have LVM. Your /boot is on sda2 on ext2, correct?

dmpm said:
4c8d35cd-3cbd-4a37-8b0f-67060928dcdd is the UUID for /dev/sda2, which is the boot partition.

I should not read line by line.

But that's ext2, correct?

dmpm said:
After removing the USB and rebooting, it's working again and prompts for the cryptlvm password and accepts it if I type it in directly.

I am bit lost now, does this work now simply after you rebuilt initramfs, or only sometimes?

dmpm said:
However, dropbear still gives an error after I type in the password 'Error: Timeout reached while waiting for PID 227' but I can still type it in on the server after that and it boots.

Yeah this one would need separate troubleshooting.

dmpm said:
Running efibootmgr -v in proxmox shows:

BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,0001
Boot0000* proxmox HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\PROXMOX\SHIMX64.EFI)
Boot0001* UEFI OS HD(1,GPT,db90aff9-80c2-459c-8c65-9b4c8191450b,0x800,0x64000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

So I guess /dev/sda is HD1 as far as efibootmgr is concerned, but HD0 in grub.

That looks alright to me.

dmpm · 2024-10-15T23:54:42+0200

esi_y said:
I think that's benign, it can't read the format kernel provides it in last time i checked.

I am not familiar with the guide to the dot, but I am pretty sure this was just emphasizing that you have LUKS on GPT partition in which you then have LVM. Your /boot is on sda2 on ext2, correct?

I should not read line by line. But that's ext2, correct?

/boot is on /dev/sda2 yeah, but it's ext4 not ext2.

esi_y said:
I am bit lost now, does this work now simply after you rebuilt initramfs, or only sometimes?

I've rebooted a couple of times after rebuilding initramfs and its working, but it worked several times when I tested it yesterday, so I'm not certain if it will keep working.

esi_y said:
Yeah this one would need separate troubleshooting.

yeah, I'll see if I can get mandos working in the meantime

esi_y said:
That looks alright to me.

Cool, thanks for your help.

esi_y · 2024-10-15T23:57:03+0200

dmpm said:
Cool, thanks for your help.

I did not really help you, I just watched if this will work already on Bookworm after all.

I will want to try this later on myself, but you were too fast for my pacing.

Search

Search

[TUTORIAL] Adding Full Disk Encryption to Proxmox

dmpm

New Member

dmpm

New Member

dmpm

New Member

esi_y

Renowned Member

dmpm

New Member

dmpm

New Member

esi_y

Renowned Member

dmpm

New Member

dmpm

New Member

esi_y

Renowned Member

dmpm

New Member

esi_y

Renowned Member

dmpm

New Member

dmpm

New Member

esi_y

Renowned Member

dmpm

New Member

esi_y

Renowned Member