[SOLVED] Reproducible Proxmox Crash on Kernel 6.8.4-2-pve: KVM force termination renders Web UI, SSH, and all running VMs Unresponsive

[fiona]

Hi,

I also have the 6.5.13-5-pve kernel pinned for now. Latest 6.8.8.2 is now showing random reboots

is there anything in the logs? If not, you might want to try running journalctl -f from another host via SSH, maybe there are logs that cannot persisted to disk.

 

[antreos]

Hi, to add to what @fiona already wrote: In your first message in this thread, you reported a NULL pointer dereference with RIP pointing to blk_flush_complete_seq:

Apr 27 13:42:52 prxmx kernel: BUG: kernel NULL pointer dereference, address: 0000000000000008
[...]
Apr 27 13:42:52 prxmx kernel: RIP: 0010:blk_flush_complete_seq+0x291/0x2d0

This issue in particular should be fixed in kernel 6.8.8-1 (and higher). See [1] for more details. Please test the newer kernel and report back, especially if you still see crashes.

[1] https://forum.proxmox.com/threads/145760/page-8#post-674842

Thank you very much, Friedrich. Especially since you seem to be "one of our developers" [1] that got into that rabbit hole. I've successfully upgraded to version 6.8.8-2 using Fiona's method. So far, I haven't been able to reproduce the freezing issue. I'll keep a close eye on the system's stability and performance to see if it remains stable.

 

[intrax]

There is never a kernel that works for each possible hardware configuration. There always are kernel regressions, it's just a too big piece of software written in a very old language. Monitoring the forum (in particular the kernel or release announcement threads) for issues reported by people with similar hardware is good, but best is to test if the new kernel works for your setup when you have a maintenance window. Nobody else will be able to guarantee that 100% up front. We do try to minimize issues, but we can only test on the hardware we have and backport fixes that can be identified.

I acknowledge that but hey this thread: https://forum.proxmox.com/threads/random-6-8-4-2-pve-kernel-crashes.145760/
has 9 pages of random reboots ! Still holding ff ftm...

 

[antreos]

Update: Two weeks now with no crashes/reboots or anything weird on 6.8.8-2-pve. Looks great so far, will continue monitoring!

 

[jamesharr]

Hi,
there is a build of kernel 6.8.8 available on the no-subscription repository since two weeks. That should resolve some of the issues people have reported with 6.8.4. If you have a subscription, you can temporarily enable the repository, run apt update, install the new kernel with apt install proxmox-kernel-6.8 and then disable the repository again and run apt update again.

@fiona , can you confirm if this package is now in the subscription repo? I'm looking and I see 6.8.8-2 on both a production (support) servers and one of our lab servers (community) when I run `apt changelog proxmox-kernel-6.8`, and I see 6.8.8-2 available. So I think it's available now, but I want to make sure I'm reading it right.

We had 2/5 servers in a cluster hard-lock in rapid succession (logs stopped, console unresponsive) after about 60 days of uptime.
- Proxmox 8.2.0
- Kernel 6.8.4-2-pve
- Ceph 18.2.2-pve1
- CPU EPYC 7543

Unless we hear otherwise, we're probably going to try 6.8.8-2 before we try rolling back to a 6.5 kernel.

 

[fiona]

Hi,

@fiona , can you confirm if this package is now in the subscription repo? I'm looking and I see 6.8.8-2 on both a production (support) servers and one of our lab servers (community) when I run `apt changelog proxmox-kernel-6.8`, and I see 6.8.8-2 available. So I think it's available now, but I want to make sure I'm reading it right.

yes, proxmox-kernel-6.8.8-2-pve is available on the enterprise/subscription repository since the 9th of July.