Root on encrypted ZFS boot issue suddenly

DAVe3283

DAVe3283

Member
Proxmox Subscriber
Mar 23, 2023
8
2
8
Boise, ID
I rebooted one of my nodes to get a new kernel, and on startup I was dropped into initramfs busybox. It is similar to this: https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox

I am running (unsupported, I know) an encrypted ZFS rpool:
Code: 
root@ThreadReaper:~# zfs get encryption,encryptionroot,keylocation,keyformat,keystatus rpool/ROOT rpool/ROOT/pve-1
NAME              PROPERTY        VALUE        SOURCE
rpool/ROOT        encryption      aes-256-gcm  -
rpool/ROOT        encryptionroot  rpool/ROOT   -
rpool/ROOT        keylocation     prompt       local
rpool/ROOT        keyformat       passphrase   -
rpool/ROOT        keystatus       available    -
rpool/ROOT/pve-1  encryption      aes-256-gcm  -
rpool/ROOT/pve-1  encryptionroot  rpool/ROOT   -
rpool/ROOT/pve-1  keylocation     none         default
rpool/ROOT/pve-1  keyformat       passphrase   -
rpool/ROOT/pve-1  keystatus       available    -

Up until this reboot, so somewhere between mid-July and today, it would simply prompt me for the passphrase Enter passphrase for 'rpool/ROOT': then continue booting. But now I get the (initramfs) prompt.

I was able to figure out how to get the system booted:

Bash: 
zpool import -l -R /root rpool
# Key load error: Failed to open key material file: No such file or directory
# Enter passphrase for 'rpool/ROOT': ******
# 1 / 2 keys successfully loaded
exit
I believe the key failure is for rpool/data, which does automatically unlock just fine later in the boot with a keyfile stored in rpool/ROOT/pve-1. I could probably just mount rpool/ROOT/pve-1 and avoid the error message, but I don't want to reboot again to find out at 00:26 in the morning.

While debugging, I tried the following with no change in this new behavior:
  • Add rootdelay=30 to the kernel command line. This just had it sit there for ~30 seconds before dropping me to the busybox shell.
  • Boot the previous working kernel (proxmox-kernel-7.0.14-4-pve-signed). This did the same strange new behavior, so not a kernel issue.
  • Install an even fresher kernel (proxmox-kernel-7.0.14-17-pve-signed). Same behavior.
I have not rebooted the other nodes yet, so I don't know if it is just this one server or all of them. Since I have to manually interact with the servers to boot them no matter what, this is not a huge problem, but it is way less convenient to have to remember the commands above to get my server to boot. Did something change that broke the previously-working-but-unsupported encrypted ZFS root? Or did something in my setup get corrupted?
 
One strange thing I noticed is in /boot/grub/grub.cfg, the kernel argument is root=ZFS=/ROOT/pve-1 instead of root=ZFS=rpool/ROOT/pve-1:
Code: 
menuentry 'Proxmox VE GNU/Linux' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-/dev/nvme4n1p3_/dev/nvme5n1p3' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod part_gpt
        echo    'Loading Linux 7.0.14-17-pve ...'
        linux   /ROOT/pve-1@/boot/vmlinuz-7.0.14-17-pve root=ZFS=/ROOT/pve-1 ro
        echo    'Loading initial ramdisk ...'
        initrd  /ROOT/pve-1@/boot/initrd.img-7.0.14-17-pve
}

It seems to have dropped the rpool out of the command line. Interestingly, rpool is not missing in the same file on a different node, implying something is wrong with how grub.cfg is generated on this particular node. It actually looks like it is missing all the command line arguments. This is how the file looks on a different node:
Code: 
menuentry 'Proxmox VE GNU/Linux' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-/dev/sda3_/dev/sdb3' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod part_gpt
        echo    'Loading Linux 7.0.14-19-pve ...'
        linux   /ROOT/pve-1@/boot/vmlinuz-7.0.14-19-pve root=ZFS=/ROOT/pve-1 ro  root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet
        echo    'Loading initial ramdisk ...'
        initrd  /ROOT/pve-1@/boot/initrd.img-7.0.14-19-pve
}
 
Last edited:
could you post "pveversion -v" and the contents of /var/log/apt/term.log referencing the last refresh of the grub config (from both nodes)?
 
Last edited:
  • Like
Reactions: Stoiko Ivanov
ThreadReaper (node with issue)
Code: 
root@ThreadReaper:~# pveversion -v
proxmox-ve: 9.2.0 (running kernel: 7.0.14-17-pve)
pve-manager: 9.2.20 (running version: 9.2.20/49318c671b82f31e)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-17-pve-signed: 7.0.14-17
proxmox-kernel-7.0: 7.0.14-17
proxmox-kernel-7.0.14-16-pve-signed: 7.0.14-16
proxmox-kernel-7.0.14-4-pve-signed: 7.0.14-4
proxmox-kernel-6.17: 6.17.13-21
proxmox-kernel-6.17.13-21-pve-signed: 6.17.13-21
amd64-microcode: 3.20251202.1~bpo13+1
ceph: 19.2.6-pve4
ceph-fuse: 19.2.6-pve4
corosync: 3.1.10-pve3
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve3
ifupdown2: 3.3.0-1+pmx12
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-7
libproxmox-acme-perl: 1.7.2
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.2
libpve-apiclient-perl: 3.4.3
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.2.2
libpve-guest-common-perl: 6.0.5
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.7
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.10
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.5-1
proxmox-backup-file-restore: 4.2.5-1
proxmox-backup-restore-image: 1.0.0
proxmox-enterprise-support-keyring: 1.1
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.10
pve-cluster: 9.1.6
pve-container: 6.1.14
pve-docs: 9.2.12
pve-edk2-firmware: 4.2026.08-1
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.6
pve-firmware: 3.18-6
pve-ha-manager: 5.2.5
pve-i18n: 3.10.0
pve-qemu-kvm: 11.0.3-3
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.8
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.4-pve1

Wrex (alternate node):
Code: 
root@Wrex:~# pveversion -v
proxmox-ve: 9.2.0 (running kernel: 7.0.2-6-pve)
pve-manager: 9.2.20 (running version: 9.2.20/49318c671b82f31e)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-19-pve-signed: 7.0.14-19
proxmox-kernel-7.0: 7.0.14-19
proxmox-kernel-7.0.14-12-pve-signed: 7.0.14-12
proxmox-kernel-7.0.14-8-pve-signed: 7.0.14-8
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
proxmox-kernel-6.17: 6.17.13-21
proxmox-kernel-6.17.13-21-pve-signed: 6.17.13-21
ceph: 19.2.6-pve4
ceph-fuse: 19.2.6-pve4
corosync: 3.1.10-pve3
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve3
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-7
libproxmox-acme-perl: 1.7.2
libproxmox-backup-qemu0: 2.0.3
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.2
libpve-apiclient-perl: 3.4.3
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.2.2
libpve-guest-common-perl: 6.0.5
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.7
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.10
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.6-1
proxmox-backup-file-restore: 4.2.6-1
proxmox-backup-restore-image: 1.0.0
proxmox-enterprise-support-keyring: 1.1
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.10
pve-cluster: 9.1.6
pve-container: 6.1.14
pve-docs: 9.2.12
pve-edk2-firmware: 4.2026.08-1
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.6
pve-firmware: 3.18-6
pve-ha-manager: 5.2.5
pve-i18n: 3.10.0
pve-qemu-kvm: 11.0.3-3
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.8
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.4-pve1
 

Attachments

You must log in or register to reply here.
Top Bottom
Back