Proxmox Kernel 6.8.12-2 Freezes (again)

[marcioindau]

On my AMD-cpu? seems like a good idea! ;-)

amd also have microcode update...

 

[thj]

Updating micrcode would solve this freezes? I have them too now. When server was always stable until latest update.

My CPU is actually the same as from this guy:

https://www.reddit.com/r/linux/comments/15xvpfg/updating_your_amd_microcode_in_linux/

But instructions are too complicated for me to follow. Do not even know if I have the latest microcode installed:

vendor_id       : AuthenticAMD
cpu family      : 23
model           : 96
model name      : AMD Ryzen 7 4800H with Radeon Graphics
stepping        : 1
microcode       : 0x8600106

So what is the official workaround? There are two options from this page https://www.thomas-krenn.com/de/wiki/Known_Issues_Proxmox_VE_8.2.

Downgrade kernel to 6.5 or make changes for 6.8. But for this second option I do not even have the file /etc/kernel/cmdline.

So what do you suggest? I had two freezes in last 3 days. Kernel downgrade?

 

[boxcee]

Not 100% sure that it is related, but I had a crontab entry for setting the CPU governor to `performance` after every reboot. I removed the entry yesterday and set the governor to `powersave` instead. So far it has been running without interruptions.

I used a tteck helper script for setting up the crontab. This one: https://github.com/tteck/Proxmox/blob/main/misc/scaling-governor.sh.

Edit: I also have the microcode for Intel installed.

 

[thj]

Not 100% sure that it is related, but I had a crontab entry for setting the CPU governor to `performance` after every reboot. I removed the entry yesterday and set the governor to `powersave` instead. So far it has been running without interruptions.

I used a tteck helper script for setting up the crontab. This one: https://github.com/tteck/Proxmox/blob/main/misc/scaling-governor.sh.

Edit: I also have the microcode for Intel installed.

I am not using this crontab at all and CPU governor is set to default. Never changed it.

# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance

How did you update microcode?

 

[boxcee]

How did you update microcode?

I used this script: https://github.com/tteck/Proxmox/blob/main/misc/microcode.sh.

 

[thj]

I used this script: https://github.com/tteck/Proxmox/blob/main/misc/microcode.sh.

Tried it myself. Didn't do anyhting:

# journalctl -k | grep -E "microcode" | head -n 1
Oct 04 08:22:21 proxmox kernel: Zenbleed: please update your microcode for the most optimal fix

I did get disk not found error on reboot:

https://pve.proxmox.com/wiki/Recover_From_Grub_Failure#Recovering_from_grub_"disk_not_found"_error_when_booting_from_LVM

Had to make bootable USB to solve it.

# journalctl -k | grep -E "microcode" | head -n 1
Oct 04 08:22:21 proxmox kernel: Zenbleed: please update your microcode for the most optimal fix

Also I am having issues with network on reboot. It takes a while to start working and I need to unplug the ethernet in and out a couple of times. I have Realtek RTL8111/8168/8411 with which I had issues in the past with different kernel version. It seems the problem is back with 6.8.

So I still live in fear of the next freeze when I will not be at home that happens. Should I pin it back to 6.5 kernel?

 

[DevUser]

the only thing that has helped me is to delete all the lxc with kernel different from PVE(ubuntu)... at this moment I am only using debian 12 in all my lxc and so far, it has not crashed once.



One of the deleted lxc was ubuntu with mysql server.
the rest were simple web applications inside ubuntu 22.04

 

[csawyer]

Where is Proxmox Support? With all respect to those who have contributed to this thread, there is no reasonable solution i can detect that I would attempt on a subscribed production cluster. It annoys me that Proxmox is leaving it to community to sort out their bugs and it doesn't give me confidence that any advertised proxmox-kernel update should be attempted on a stable cluster

 

[gkovacs]

It may be totally unrelated, but we hade sporadic kernel freezes for a long time (happening every few days to every few weeks) on many servers since Proxmox 6 and then 7, using several different kernel versions. We tried many things, but the one thing that ended all freezes was disabling X2APIC both in BIOS/UEFI and grub.

GRUB_CMDLINE_LINUX_DEFAULT="quiet consoleblank=0 nox2apic"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs consoleblank=0 nox2apic"

We are currently running only dual Intel Xeons (several different generations) with kernel 6.8.12-1, and so far no unexplainable crashes since disabling X2APIC last year.

 

[Ancoor]

Interesting since 6.8.12-X my N305 server also hungs, no output on HDMI or keyboard.

I had severe problems with my N305 host since I upgraded the kernel.... I actually thought the hardware was kaputt and bought new barebone... (not yet arrived). Now I read this and it almost make me cry... I'm having spontaneous reboots/resets, without any logs or anything.

 

[Ancoor]

I cannot really deal with my spontaneous reboots. As there is no working workaround from Proxmox for the kernel, I will need to move away from the repository and install kernel 6.11.

Is there at minimum an estimated ETA for Proxmox new kernel release?

 

[Decco1337]

Updating micrcode would solve this freezes? I have them too now. When server was always stable until latest update.

My CPU is actually the same as from this guy:

https://www.reddit.com/r/linux/comments/15xvpfg/updating_your_amd_microcode_in_linux/

But instructions are too complicated for me to follow. Do not even know if I have the latest microcode installed:

vendor_id       : AuthenticAMD
cpu family      : 23
model           : 96
model name      : AMD Ryzen 7 4800H with Radeon Graphics
stepping        : 1
microcode       : 0x8600106

So what is the official workaround? There are two options from this page https://www.thomas-krenn.com/de/wiki/Known_Issues_Proxmox_VE_8.2.

Downgrade kernel to 6.5 or make changes for 6.8. But for this second option I do not even have the file /etc/kernel/cmdline.

So what do you suggest? I had two freezes in last 3 days. Kernel downgrade?

Unfortunately the things at Thomas Krenn didn’t help for me. Also upgraded to 6.11 manually and still have reboots.

 

[Decco1337]

Where is Proxmox Support? With all respect to those who have contributed to this thread, there is no reasonable solution i can detect that I would attempt on a subscribed production cluster. It annoys me that Proxmox is leaving it to community to sort out their bugs and it doesn't give me confidence that any advertised proxmox-kernel update should be attempted on a stable cluster

Since many companies switched to Proxmox the Proxmox staff has a lot to do with migrations and the support for this companies. Unfortunately I have the feeling you just get support from them by buying a license… the golden times are over :/

 

[sub2o5]

Since many companies switched to Proxmox the Proxmox staff has a lot to do with migrations and the support for this companies. Unfortunately I have the feeling you just get support from them by buying a license… the golden times are over :/

You are also using the B650D4U? The board seems to be the problem. Many people switched to the H13SAE-MF from supermicro and the problem with sudden reboots (mine) has been solved.

Really disappointed by asrock and the seller seems to not want to take the board back. Having the third exchange board and don't trust it anymore. The X570D4U (AM4) on the other hand is rock solid!

 

[Decco1337]

You are also using the B650D4U? The board seems to be the problem. Many people switched to the H13SAE-MF from supermicro and the problem with sudden reboots (mine) has been solved.

Really disappointed by asrock and the seller seems to not want to take the board back. Having the third exchange board and don't trust it anymore. The X570D4U (AM4) on the other hand is rock solid!

Yep using this board but also with other boards this issues appears. Also with Proxmox 7 I had no issues. So I don’t think it’s the board The board has some issues, but you can fix and then this issue is solved.

I also know people using Supermicro and older Intel systems with the same issues since Proxmox 8.

 

[sub2o5]

Yep using this board but also with other boards this issues appears. Also with Proxmox 7 I had no issues. So I don’t think it’s the board The board has some issues, but you can fix and then this issue is solved.

I also know people using Supermicro and older Intel systems with the same issues since Proxmox 8.

After my second board started randomly rebooting after several weeks, i had enough.
Also other people mention that even H5 REV 4.01 has the Post-Code-00-Problem...
I dont know

 

[Decco1337]

After my second board started randomly rebooting after several weeks, i had enough.
Also other people mention that even H5 REV 4.01 has the Post-Code-00-Problem...
I dont know

I’ll keep that in mind, thanks! Just in time I wrote my message I got a server with that board which was unresponsive. Needed to reset the IPMI otherwise a reboot didn’t work haha…

But also using this board for customers with Proxmox 7 and other OS without any issues. Strange…. All boards and other boards with Proxmox 8 have reboots or hangs. Doesn’t matter which kernel you take (currently using kernel 6.11)

 

[Decco1337]

Seems it has something to do with PCIe / NICs. I have a server which crashes constantly so I decided to move the VMs away to another host. And when I start the migration (10 Gbps) the server crashes.

Currently using Mellanox CX3 Pro. Anyone else which had a similar issue?

PS: Never had this before while migrating btw

 

[sub2o5]

Currently using Mellanox CX3 Pro. Anyone else which had a similar issue?

Interesting. I'm using the same (40G) with the system.
No problem with X570, only with B650

 

[sub2o5]

I installed the CX3 Pro on Juli, 6th. Reboots occured on August, 26. So i don't see a direct relation there.