Help with Boot/ZFS Issue

[tokenwizard]

I have a server running Proxmox 8 which was running 7 and I have had functioning great for over a year.

Over that time, I have gone through the process of upgrading several of the disks from 4TB to 8TB ones.
Each time, I replicated the partition table of the old disk onto the new one using sgdisk and then expanded the third partition to use in the ZFS pool.
I also used proxmox-boot-tool to update the bootloader on the new disk.

After each disk replacement, the system could be rebooted and would always boot fine.

Today, I updated the system, including installing kernels 8.8.8-2 and 8.8.8-4 and now my system will not boot at all.

BIOS is set to GRUB boot mode as it always has been (and worked fine all this time).

So far, what I have tried...
1) boot into UEFI mode, chroot into the proxmox installation, and:

proxmox-boot-tool format <ESP partition 2>

for each of the 5 disks included int he ZFS pool

proxmox-boot-tool init <ESP partition 2>

for each of the 5 disks

2) boot into GRUB mode and do the same steps above.

I have tried to format/init the first partition on each disk while booted into GRUB mode, but it complains that the partition is too small.

I don't understand why I suddenly can't boot. It seems like I'm missing something silly, but I'm not sure what else to try. I can import my zpool and access it in the chroot and

zpool status

shows no errors.

"Normal" boot with the BIOS set to GRUB mode, gets stuck here, before presenting the GRUB Boot Menu:


And if I boot to the UEFI Boot Options, all the options for the system are greyed out except the Network ones:


If I switch the BIOS to UEFI Boot Mode, and let it try to boot normally, It fails here:

 

[tokenwizard]

Additional Details:

Booted into UEFI Boot Mode and chroot'd into the proxmox install, bootctl shows boot partitions on each of the 5 disks:


efibootmgr shows several entries:


proxmox-boot-tool status/refresh:

 

[tokenwizard]

For more info, here is the partition layout of all 5 disks included in the zfs pool. They all have this same partition scheme (sda, sdb, sdc, sde, sdf). Each one has an EFI/ESP partition #2.


Within the CHROOT environment, you can see that the EFI partition is populated:









Apologies for all the screenshots. I'm working in an iDRAC console and can't copy/paste the console output.

 

[tokenwizard]

Ok, I think I am getting closer now.
I ran through the steps here: https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool

And switched by BIOS back to GRUB/BIOS mode. Now the system does boot up but hits initramfs. I'm thinking the ZFS root is not getting set or maybe not getting mounted early enough for the boot process?

UPDATE: after a

(initramfs) zpool import -f
(initramfs) exit

I am back in business and running kernel 6.8.8-4!