Upgrade to PVE 7.0 failed. "Failed to load ZFS Modules. Manually load the modules and exit"

[piexil]

Hi all, upgraded from 6 to 7 followng the exact procedure here: https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0

After doing apt-dist upgrade and rebooting my nodes they're all stuck at the initramfs screen with the following message
"Failed to load ZFS Modules. Manually load the modules and exit"

Only problem, I have no USB input. The USB keyboard works in the bios but compltley stops working on this screen. Theres also a lot of printouts from various kernel modules saying "failed to validate module [module name] BTF: -22"

Tried proxmox rescue boot and it said "could not find root harddrive partition" or so. So i loaded debug mode and am able to get a chroot from there.

Any suggestions? I tried apt install pve-kernel-5.11 --reinstall, update-initramfs -u and neither helped.

 

[piexil]

OK
I was able to fix it by doing the following.
Boot to debug mode installer, press ctrl-d at first terminal to get passed initrd.

Then at second debug terminal do
$ zpool import -f rpool $ zfs set mountpoint=/mnt rpool/ROOT/pve-1 <zfs part will now be mounted at /mnt> $ mount -t proc /proc /mnt/proc $ mount --rbind /dev /mnt/dev $ mount --rbind /sys /mnt/sys $ chroot /mnt /bin/bash (chroot)$ source /etc/profile (chroot)$ ifdown vmbr0 && ifup vmbr0 (to get internet working, may be able to skip on your setup) (chroot)$ apt purge pve-kernel-5.11 (NOTE this will remove proxmox ve package) (chroot)$ apt install pve-headers pve-headers-5.11 pve-kernel-5.11 proxmox-ve (chroot)$ proxmox-boot-tool refresh (chroot)$ exit $ umount -l /mnt $ zfs unmount rpool $ zfs set mountpoint=/ rpool/ROOT/pve-1 $ exit <reboot>

 

[piexil]

I did this across 6 nodes, all updated and working again. For some reason on three of them I had to zfs import -f rpool at the initramfs terminal and the other 3 I didn't.

Still no idea how or why this happened in the first place. Any ideas on how to avoid in the future?