pve7to8 cluster upgrade stuck on kernel 5.15.131-2-pve

[virutal_manifest]

I'm mostly successfully upgraded from pve7to8 everything seems to work except all 3 nodes in cluster are stuck on and old kernel:

pve7to8 --full
# WARN: proxmox-boot-tool is used for bootloader configuration in uefi mode but the separate systemd-boot package is not installed, initializing new ESPs will not work until the package is installed

#Current state after upgrade

uname -r
5.15.131-2-pve

pveversion
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 5.15.131-2-pve)

proxmox-boot-tool kernel list 
Manually selected kernels:
None.

Automatically selected kernels:
5.15.131-2-pve
6.5.11-7-pve

I've tried re-installing proxmox-ve and pve-kernel and rebooting with no change.

apt reinstall proxmox-ve
apt reinstall pve-kernel-6.5.11-7-pve

Does anyone have any suggestions to upgrade the kernel? Any help is greatly appreciated.

I'm not sure if it's related but after confirming my pve7 install was using EFI I installed system-boot after the update.
efibootmgr -v
apt install systemd-boot

 

[virutal_manifest]

Looks like I didn't read carefully enough. I installed `systemd-boot` thinking I needed it, but I'm not using ZFS on root.

```shell
df -Th
/dev/mapper/pve-root ext4 197G 36G 152G 19% /

apt --reinstall install proxmox-kernel-helper

Preparing to unpack .../proxmox-kernel-helper_8.1.0_all.deb ...
Unpacking proxmox-kernel-helper (8.1.0) over (8.1.0) ...
Setting up proxmox-kernel-helper (8.1.0) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.5.11-7-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
/dev/disk/by-uuid/714B-709F contains no grub directory - skipping
```

 

[virutal_manifest]

If anyone else experiences the same. I have one 1/3 nodes booting to 6.5.11-7-pve now. Steps to fix the issue.
- Remove systemd-boot
- Update the boot order

efibootmgr -v
BootCurrent: 0001
Timeout: 1 seconds
BootOrder: 0001,0002,0000
Boot0000* debian        HD(1,GPT,7881a5cb-2ab4-4c51-9d30-dea59a4abe5a,0x800,0x100000)/File(\EFI\debian\shimx64.efi)
Boot0001* Linux Boot Manager    HD(2,GPT,36feba97-d4f6-4855-bd3b-42b38f3e4bd0,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot0002* proxmox       HD(2,GPT,36feba97-d4f6-4855-bd3b-42b38f3e4bd0,0x800,0x100000)/File(\EFI\proxmox\grubx64.efi)

Installing systemd-boot seems to have been the issue. The system was booting to systemd boot manager instead of grub.

Steps taken:
apt remove systemd-boot efibootmgr -o 0002,0001,0000

root@pve1:~# lsblk -o +FSTYPE NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS FSTYPE sda 8:0 0 1.8T 0 disk ├─sda1 8:1 0 1007K 0 part ├─sda2 8:2 0 512M 0 part /boot/efi vfat └─sda3 8:3 0 1.8T 0 part LVM2_member ├─pve-swap 252:0 0 8G 0 lvm [SWAP] swap ├─pve-root 252:1 0 200.1G 0 lvm / ext4 ├─pve-data_tmeta 252:2 0 4.6G 0 lvm │ └─pve-data-tpool 252:5 0 1.6T 0 lvm │ ├─pve-data 252:6 0 1.6T 1 lvm findmnt / TARGET SOURCE FSTYPE OPTIONS / /dev/mapper/pve-root ext4 rw,relatime,errors=remount-ro


Looks like proxmox-boot-tool init /dev/sda2 grub would have worked as well, while also integrating kernel with secure boot.