Proxmox Kernel 6.8.4.3 upgrade problem (kernel panic)

[GUIn00b]

I tried updating my node and when it rebooted it went into kernel panic. I was able to reboot and choose a previous kernel and it's running fine.

I want to purge the failed 6.8.4.3 stuff from my system and attempt the upgrade again. I'm not sure how to do this, and the things I tried said scary things like "YOU'RE TRYING TO REMOVE PROXMOX COMPLETELY" and stuff lol

 

[leesteken]

You cannot remove the latest kernel, it is required by Proxmox (according to the package dependencies). Just pin the previous (or another working) kernel in proxmox-boot-tool, so Proxmox does not start the problematic kernel: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_kernel_pin

 

[GUIn00b]

Is there any way to reinstall the upgrade or should I just wait until a newer version is released?

 

[leesteken]

Is there any way to reinstall the upgrade

I don't understand what you are asking. If apt dist-upgrade does not give any errors, then you're up to date. What are you trying to fix with reinstalling a specific package?

or should I just wait until a newer version is released?

It looks like lots of people are having issues with kernel 6.8 (according to threads on this forum) and are indeed waiting for the next update.

 

[GUIn00b]

Do you see in the above screenshot where it says "4 not fully installed or removed" ? So I'm concerned that I have some sort of partial package installs/upgrades or something.

Would you suggest that, since the node is operational using the previous GRUB entry (in my case 6.5.13-5), should I just leave it as is?

One more question. When I go into the webGUI, click on the node, go to Summary, then click "Package Versions", it shows this list:

proxmox-ve: not correctly installed (running kernel: 6.5.13-5-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.2
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.2-1
proxmox-backup-file-restore: 3.2.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.6
pve-container: 5.1.10
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.7
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2

Notice it says "proxmox-ve: not correctly installed" on the first line? Is that ok? Here's a screenshot too:

 

[leesteken]

Do you see in the above screenshot where it says "4 not fully installed or removed" ? So I'm concerned that I have some sort of partial package installs/upgrades or something.

Would you suggest that, since the node is operational using the previous GRUB entry (in my case 6.5.13-5), should I just leave it as is?

Run apt dist-upgrade. EDITS: That will not change the booting kernel if you pinned it. And I would remove kernel 6.2 since it is not maintained.
MORE-EDITS: I assumes the previous problems were from running a remove but if apt dist-ugprade is giving these problems, make sure you don't have the Debian kernel installed. It's really hard to advise without seeing the whole output of apt dist-upgrade (and knowing what you changed before to cause this).

 

[leesteken]

If this is not resolved yet, can you show the whole output of apt update && apt dist-upgrade (in CODE tags)?

 

[GUIn00b]

Yes, my apologies I've been away unexpectedly. I will update this throughout today. Thank you for your help!!

 

[GUIn00b]

Here finally (sorry again for the delay/radio silence; life is full of "plot twists" these days!)

root@pve:~# apt update && apt dist-upgrade
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 http://security.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]       
Ign:4 https://download.webmin.com/download/newkey/repository stable InRelease 
Get:5 http://security.debian.org/debian-security bookworm-security/main amd64 Packages [160 kB]
Hit:6 https://download.webmin.com/download/newkey/repository stable Release   
Get:7 http://security.debian.org/debian-security bookworm-security/main Translation-en [96.4 kB]
Get:8 http://security.debian.org/debian-security bookworm-security/contrib amd64 Packages [644 B]
Get:9 http://security.debian.org/debian-security bookworm-security/contrib Translation-en [372 B]
Get:10 http://deb.debian.org/debian bookworm/main amd64 Packages [8,786 kB]   
Hit:12 http://download.proxmox.com/debian/pve bookworm InRelease               
Get:13 http://deb.debian.org/debian bookworm/main Translation-en [6,109 kB]   
Get:14 http://deb.debian.org/debian bookworm/contrib amd64 Packages [54.1 kB]
Get:15 http://deb.debian.org/debian bookworm/contrib Translation-en [48.7 kB]
Get:16 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.8 kB]
Get:17 http://deb.debian.org/debian bookworm-updates/main Translation-en [16.0 kB]
Get:18 http://deb.debian.org/debian bookworm-updates/contrib amd64 Packages [768 B]
Get:19 http://deb.debian.org/debian bookworm-updates/contrib Translation-en [408 B]
Fetched 15.5 MB in 5s (3,386 kB/s)                                   
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libarchive13 libgstreamer-plugins-base1.0-0 proxmox-backup-client
  proxmox-backup-file-restore
4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
4 not fully installed or removed.
Need to get 7,200 kB of archives.
After this operation, 31.7 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://security.debian.org/debian-security bookworm-security/main amd64 libarchive13 amd64 3.6.2-1+deb12u1 [343 kB]
Get:2 http://security.debian.org/debian-security bookworm-security/main amd64 libgstreamer-plugins-base1.0-0 amd64 1.22.0-3+deb12u2 [934 kB]
Get:3 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 proxmox-backup-client amd64 3.2.3-1 [2,960 kB]
Get:4 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 proxmox-backup-file-restore amd64 3.2.3-1 [2,964 kB]
Fetched 7,200 kB in 3s (2,529 kB/s)                     
Reading changelogs... Done
(Reading database ... 156925 files and directories currently installed.)
Progress: [  4%] [######......................................................................................................................................................]
Unpacking libarchive13:amd64 (3.6.2-1+deb12u1) over (3.6.2-1) ...
Preparing to unpack .../libgstreamer-plugins-base1.0-0_1.22.0-3+deb12u2_amd64.deb ...
Unpacking libgstreamer-plugins-base1.0-0:amd64 (1.22.0-3+deb12u2) over (1.22.0-3+deb12u1) ...
Preparing to unpack .../proxmox-backup-client_3.2.3-1_amd64.deb ...
Unpacking proxmox-backup-client (3.2.3-1) over (3.2.2-1) ...
Preparing to unpack .../proxmox-backup-file-restore_3.2.3-1_amd64.deb ...
Unpacking proxmox-backup-file-restore (3.2.3-1) over (3.2.2-1) ...
Setting up proxmox-backup-file-restore (3.2.3-1) ...
Updating file-restore initramfs...
12781 blocks
Setting up libarchive13:amd64 (3.6.2-1+deb12u1) ...
Setting up libgstreamer-plugins-base1.0-0:amd64 (1.22.0-3+deb12u2) ...
Setting up proxmox-kernel-6.8.4-3-pve (6.8.4-3) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/dkms 6.8.4-3-pve /boot/vmlinuz-6.8.4-3-pve
dkms: running auto installation service for kernel 6.8.4-3-pve.
Sign command: /lib/modules/6.8.4-3-pve/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub

Building module:
Cleaning build area...
make -j48 KERNELRELEASE=6.8.4-3-pve KDIR=/lib/modules/6.8.4-3-pve/build...(bad exit status: 2)
Error! Bad return status for module build on kernel: 6.8.4-3-pve (x86_64)
Consult /var/lib/dkms/vendor-reset/0.1.1/build/make.log for more information.
Error! One or more modules failed to install during autoinstall.
Refer to previous errors for more information.
dkms: autoinstall for kernel: 6.8.4-3-pve failed!
run-parts: /etc/kernel/postinst.d/dkms exited with return code 11
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.8.4-3-pve.postinst line 20.
dpkg: error processing package proxmox-kernel-6.8.4-3-pve (--configure):
 installed proxmox-kernel-6.8.4-3-pve package post-installation script subprocess returned error exit status 2
Setting up proxmox-backup-client (3.2.3-1) ...
dpkg: dependency problems prevent configuration of proxmox-kernel-6.8:
 proxmox-kernel-6.8 depends on proxmox-kernel-6.8.4-3-pve-signed | proxmox-kernel-6.8.4-3-pve; however:
  Package proxmox-kernel-6.8.4-3-pve-signed is not installed.
  Package proxmox-kernel-6.8.4-3-pve is not configured yet.
  Package proxmox-kernel-6.8.4-3-pve-signed which provides proxmox-kernel-6.8.4-3-pve is not installed.

dpkg: error processing package proxmox-kernel-6.8 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-default-kernel:
 proxmox-default-kernel depends on proxmox-kernel-6.8; however:
  Package proxmox-kernel-6.8 is not configured yet.

dpkg: error processing package proxmox-default-kernel (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on proxmox-default-kernel; however:
  Package proxmox-default-kernel is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for libc-bin (2.36-9+deb12u7) ...
Errors were encountered while processing:
 proxmox-kernel-6.8.4-3-pve
 proxmox-kernel-6.8
 proxmox-default-kernel
 proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@pve:~#

 

[leesteken]

Consult /var/lib/dkms/vendor-reset/0.1.1/build/make.log for more information.

Your vendor-reset is too old for Proxmox kernel 6.8. Uninstall it, update it to the latest version and reinstall it. I explained this before in a little more detail, but I hope you can find it yourself.

 

[GUIn00b]

Your vendor-reset is too old for Proxmox kernel 6.8. Uninstall it, update it to the latest version and reinstall it. I explained this before in a little more detail, but I hope you can find it yourself.

Well, I don't know that I found any instructions posted by you specifically, but after some digging, the procedure was pretty simple in this case:

dkms remove vendor-reset/0.1.1 --all
apt update && apt dist-upgrade

Then I followed the same tutorial to pin the 6.8 grub entry that I used to pin the 6.5 entry: https://forum.proxmox.com/threads/select-default-boot-kernel.79582/post-398029

Voila! Thanks for your guidance, @leesteken it was just the right amount of "nudge" to get me to the right outcome. Big hugs!