Kernel 6.8.4-2 causes random server freezing

[eebgmbh]

My experience so far. Pinned the kernel to 6.5.
Then updated one machine, out of four, to 6.8.4-3-pve, machine has AMD Opteron(tm) Processor 6380 (2 Sockets/ 32 Cores). Running now since 37 days without crash. But this machine crashed only twice with 6.8.4-3-pve, within a week or so. The other machines (same Hardware) stalled much more often.

This might help somebody.

 

[w_bufffet]

Hi, anyone found a solution for this?

 

[tack]

Hi, anyone found a solution for this?

Yes, upgrade to at least proxmox-kernel 6.8.8-1.

 

[w_bufffet]

I

Yes, upgrade to at least proxmox-kernel 6.8.8-1.

the latest pve iso (fully updated) causes those freeze on my ryzen 3900x with x570 chipset. using the 8.1 version stopped the issue. So I will wait for the next release and see.

 

[antonin.chadima]

latest kernel Linux 6.8.8-2-pve seems to be OK
one experimental server with this kernel is running for a week and did not freeze
so i will upgrade the rest of them....

 

[BenediktS]

I am also running our small cluster on Linux 6.8.8-1 for 23 days now.
I will update our big cluster tonight.

It looks like the 6.8.8 Kernel fixes the node freezing problem for us.

 

[WoodBench]

I have freezes with 6.8.8-2-pve within 24h of restarting guests, pinning 6.5.13-5-pve have run solid for 48h now.

Proxmox VE 8.2.4, CPU Intel N6005, guests Ubuntu server 24.04

 

[antonin.chadima]

I have freezes with 6.8.8-2-pve within 24h of restarting guests, pinning 6.5.13-5-pve have run solid for 48h now.

Proxmox VE 8.2.4, CPU Intel N6005, guests Ubuntu server 24.04

I will try to restart the guests on the server with 6.8 kernel

 

[WoodBench]

Never mind, my guest froze again on 6.5.13-5-pve - I guess my problem is different.

 

[BenediktS]

This here was more about the proxmox nodes itself freezing.
If your VMs freez, you may look into this thread:

https://forum.proxmox.com/threads/what-is-kvm-desc-next-is-3-indicative-of.128739/page-3#post-678267

 

[WoodBench]

This here was more about the proxmox nodes itself freezing.
If your VMs freez, you may look into this thread:

https://forum.proxmox.com/threads/what-is-kvm-desc-next-is-3-indicative-of.128739/page-3#post-678267

I don't have those error messages, but thank you for the suggestion. I don't want to hog this post, so I've posted my own.
https://forum.proxmox.com/threads/proxmox-ve-8-2-4-random-guest-freezes.151294/

 

[antonin.chadima]

A rack of Proxmox servers on 6.8.8-2-pve kernel and so far no problem.

 

[IT4U]

A rack of Proxmox servers on 6.8.8-2-pve kernel and so far no problem.

antonin.chadima did you have any issues with 6.8.8-2 so far?

 

[antonin.chadima]

antonin.chadima did you have any issues with 6.8.8-2 so far?

It is still ok. Some very old VM's like Debian 4, 5, 6... Etc. And similar Ubuntu (Yes we have to use them) have randomly problems (with IO?) and freeze with some kernel panic... Sometimes . Not often....

But it is ok!

 

[qXt69WEV2a7fgbET]

Same Problem on all of my Genoa hosts, going to downgrade now as other possible causes have been eliminated.
The hosts just die without any sys or kernel buffer logs, not related to high load or anything like that.

The OS doesn't respond to shutdown request or keyboard input.
The affected hosts are running on 6.8.12-1-pve.

I notice buffer/cache occupancy dropping while iowait increases and then the host is gone without logging anything.
I already suspected this to be some kind of fstrim related issues as this occurs every week already which reminded we of the standard systemd-timer config of runnning fstrim once a week with a timer, but this doesn't seem to be related.

My next step would have been to exchange sata boot drives against m.2 nvme but after i saw this thread i will downgrade first.
Thanks to the community for participating that active.

 

[KorDen]

I have single-node server with Ryzen 9 5950X on X570D4U mobo and 2 NVMe drives (ZFS Mirror) and I also experience random freezes ever since 6.8 kernel (right now on 6.8.12-4-pve)
It turned out that freezes happens when server is idle even without any VMs running.
Server hangs in 5-30 minutes if I cold-start it with all VMs autostart disabled and no other load, just pinging it from my machine.
It doesn't hang for weeks if I run any continuous load (e.g. at least one core always busy) - doesn't matter if it's on host or in VM. But it'll freeze as soon as there are no load for 5-30 minutes.

Update:
6.5.13-6-pve - freezes on idle
6.8.12-4-pve - freezes on idle
6.11.0-1-pve - freezes on idle