Kernel not updating to 6.8

[luison]

We've just upgraded a server to PVE8 from Vo7. All went smooth but I am rather uncertain of why the kernel is not being updated.

proxmox-boot-tool kernel list
Manually selected kernels:
5.3.10-1-pve

Automatically selected kernels:
5.15.158-2-pve
6.8.8-4-pve

 proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
2C6E-A8ED is configured with: uefi (versions: 5.15.158-1-pve, 5.15.158-2-pve, 5.3.10-1-pve, 5.4.203-1-pve, grubx64.efi)
7885-910E is configured with: uefi (versions: 5.15.158-1-pve, 5.15.158-2-pve, 5.3.10-1-pve, 5.4.203-1-pve, grubx64.efi)
D060-1673 is configured with: uefi (versions: 5.15.158-1-pve, 5.15.158-2-pve, 5.3.10-1-pve, 5.4.203-1-pve, grubx64.efi)

 proxmox-boot-tool refresh
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
/dev/disk/by-uuid/2C6E-A8ED contains no grub directory - skipping
/dev/disk/by-uuid/7885-910E contains no grub directory - skipping
/dev/disk/by-uuid/D060-1673 contains no grub directory - skipping

No pending updates
I understand no pending reboots either:

cat: /var/run/reboot-required: No existe el fichero o el directorio

pveversion -v
proxmox-ve: 8.2.0 (running kernel: 5.15.158-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-15
proxmox-kernel-6.8: 6.8.8-4
proxmox-kernel-6.8.8-4-pve-signed: 6.8.8-4
pve-kernel-5.15.158-2-pve: 5.15.158-2
pve-kernel-5.15.158-1-pve: 5.15.158-1
pve-kernel-5.4.203-1-pve: 5.4.203-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.13-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.3
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

 

[UdoB]

Guessing: please run proxmox-boot-tool kernel and recognize that there is an "unpin" subcommand...

 

[luison]

Do not think we've ever pinned one but we'll give it a try. In any case I understand being that the case after doing that shouldn't

proxmox-boot-tool refresh

Produce a different output?
Is the "/dev/disk/by-uuid/2C6E-A8ED contains no grub directory - skipping" correct reply of the command?

 

[vesalius]

at some point 5.3.10-1-pve was pinned given the proxmox-boot-tool kernel list
output shown.

proxmox-boot-tool kernel unpin and reboot will undo this or if you want to be cautious start with proxmox-boot-tool kernel unpin --next-boot and reboot for single use trial. If all good after reboot, run proxmox-boot-tool kernel unpin and you are back on the latest kernel train.

 

[vesalius]

I should have mentioned, be prepared to potentially have some of your network interface names changed once booted into the latest kernel.

If networking is off or not working, ip a and compare the interface names with what you have with the older naming conventions in /etc/network/interfaces under kernel 5.3/systemd and make changes there as needed.

 

[luison]

at some point 5.3.10-1-pve was pinned given the proxmox-boot-tool kernel list
output shown.

proxmox-boot-tool kernel unpin and reboot will undo this or if you want to be cautious start with proxmox-boot-tool kernel unpin --next-boot and reboot for single use trial. If all good after reboot, run proxmox-boot-tool kernel unpin and you are back on the latest kernel train.

So had a try to

# proxmox-boot-tool kernel unpin
Removed /etc/kernel/next-boot-pin.
Removed /etc/kernel/proxmox-boot-pin.

But then we noticed that /etc/kernel contained a file named
proxmox-boot-manual-kernels
with the "5.3.10-1-pve" on it. So we also removed it... and rebooted.
A kernel list now showed correctly:

# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
5.15.158-2-pve
6.8.8-4-pve

But proxmox-boot-refresh maintains exactly the same result:

n# proxmox-boot-tool refresh
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
/dev/disk/by-uuid/2C6E-A8ED contains no grub directory - skipping
/dev/disk/by-uuid/7885-910E contains no grub directory - skipping
/dev/disk/by-uuid/D060-1673 contains no grub directory - skipping

which as I said I am not sure if I can ignore those grub directory errors.

But in any case

#uname -a
Linux e20home 5.15.158-2-pve #1 SMP PVE 5.15.158-2 (2024-07-26T13:11Z) x86_64 GNU/Linux

# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
2C6E-A8ED is configured with: uefi (versions: 5.15.158-1-pve, 5.15.158-2-pve, 5.3.10-1-pve, 5.4.203-1-pve, grubx64.efi)
7885-910E is configured with: uefi (versions: 5.15.158-1-pve, 5.15.158-2-pve, 5.3.10-1-pve, 5.4.203-1-pve, grubx64.efi)
D060-1673 is configured with: uefi (versions: 5.15.158-1-pve, 5.15.158-2-pve, 5.3.10-1-pve, 5.4.203-1-pve, grubx64.efi)

So the 6.8 kernel is still not being selected.
Do i need to manually "add" the kernel with proxmox-boot tool kernel add?

Also the boot menu displays kernels that have been removed, like 5.15.158-1

Not sure what I am missing. I understood that proxmox-boot-tool refresh was in charge of all this now and no need to update update-initramfs.
So where does proxmox-boot-tool kernel list get the list from?

Also noticed that the "6.8.8-4-pve" if I check for installed packages has a differente naming structure?

#dpkg --get-selections | grep pve
...
proxmox-kernel-6.8.8-4-pve-signed install
...
pve-kernel-5.15.158-2-pve install
pve-kernel-5.4.203-1-pve install

...

 

[vesalius]

The naming structure changed somewhere after the switch to the early Linux 6 kernels Promox 7 or so, which this machine locked to linux 5.3 skipped.

Not sure about the grub directory errors, but believe I have seen the same. When you're booting do you see GRUB or systemd-boot?
https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_determine_bootloader_used

If you run proxmox-boot-tool kernel pin 6.8.8-4-pve --next-boot then reboot, after that is the latest expected kernel used, do things work correctly?

dpkg --list|grep pve-kernel will list all the old kernels
dpkg --list|grep proxmox-kernel will list the more recent kernels

Then you can use apt purge pve-kernel-5.3*, apt purge pve-kernel-5.4* , etc ... to completely remove those except the ones you want to keep. Neither apt autoremove nor the normal Proxmox update process completely remove them.