Proxmox Kernel 6.8.12-2 Freezes (again)

D

Decco1337

New Member
Apr 12, 2023
28
2
3
Hello there,

I’ve updated to Proxmox Kernel 6.8.12-2 and have freezes again on my servers. With 6.8.12.-1 it was nearly stable (just one server had issues).

Unfortunately nothing in the logs, the server is dead and need to be rebooted manually.

Anyone with the same experience? Any further idea to fix that issue to 100%?

This solution https://www.thomas-krenn.com/de/wiki/Known_Issues_Proxmox_VE_8.2 fixed the issue to 99%, as I said one host had an issue. I replaced it and after a week the issue came back.

Now with the newest kernel the issue occurred on multiple updated hosts.
 
  • Like
Reactions: boxcee
I did the same tweaks from Thomas Krenn. And hope it fixes it. Perhaps now, that kernel 6.11 is out, there is a chance that 6.10 comes to Proxmox.
 
  • Like
Reactions: Decco1337
mac.linux.free said:
I did the same tweaks from Thomas Krenn. And hope it fixes it. Perhaps now, that kernel 6.11 is out, there is a chance that 6.10 comes to Proxmox.
Click to expand...
Hopefully! Honestly I am very careful right now with the kernel updates. Just upate the kernel and the hosts are going to freeze or crash again...
 
mika said:
I observed the same behavior at one customer of mine, is there a tracking ticket at https://bugzilla.proxmox.com/ already (I couldn't find any), where one could provide further information that might be useful for Proxmox team/developers?
Click to expand...
Thanks for sharing your experience! As far as I know there is no ticket yet. Unfortunately also a bit tricky as I have no issues in the logs...
 
Quiet unbelievable as we didn't observed any freezes (or other bugs) with any 6.8 kernel version on our 4 (dell t620/T430) pve's yet, actual running 6.8.12-2.
 
  • Like
Reactions: Decco1337
waltar said:
Quiet unbelievable as we didn't observed any freezes (or other bugs) with any 6.8 kernel version on our 4 (dell t620/T430) pve's yet, actual running 6.8.12-2.
Click to expand...
Which CPU type do you use for the VMs, are you running Windows VMs? I found out that it also depends which CPU type you use and if you have Windows VMs. Also nested virt made issues with the kernel. But I also had these issues with the 6.5 kernel.

I am also sure that it cannot be a hardware issue as I use multiple servers and already replaced a lot :D I also have no strange configs on the servers, just fresh installed Proxmox, the changes from Thomas Krenn (https://www.thomas-krenn.com/de/wiki/Known_Issues_Proxmox_VE_8.2) and VMs.

Also running a bit more than 4 servers, I also have a huge amount of servers which don't have any issues.
 
Last edited:
Puuh, >50 vm's/lxc's, mostly default cpu unchanged and some have host selected and yes about 7x win (server+10+11) included. There were some nested virt tests but these are off yet but don't produce errors/problems that time when running.
 
There are many threads about Proxmox VE crashes with kernel 6.8 in the last months.

We operate several thousand Proxmox VE systems and had an outage rate of approx. 10% of the systems with kernel 6.8.
The crashes are not specifically reproducible. Regardless of the load, the barebone (several HPE, Lenovo and Supermicro servers in use) or specific VMs (it needs always a group of VMs, doesn't matter which ones).
Some systems crashed several times a day, others only once a month. Some hosts had no crashes at all.
The only commonality: AMD Epyc and Genoa CPUs

We tried various configurations for ZFS, memory management, process management, power efficiency modes, KVM CPU models, with/without hyperthreading, etc. Nothing helped.

In the end, we took the 6.11 kernel from the Ubuntu repo (Proxmox is based on the Ubuntu kernel anyway) and since then the systems have been stable.
The 6.8 kernel definitely has a problem.

@proxmox Team
You should seriously consider retiring the 6.8 kernel and switching to 6.11.
 
  • Like
Reactions: Decco1337
Which makes sense as we just have intel cpus ... so we cannot see crashes with amd's on 6.8.*.
 
HostingGuy said:
There are many threads about Proxmox VE crashes with kernel 6.8 in the last months.

We operate several thousand Proxmox VE systems and had an outage rate of approx. 10% of the systems with kernel 6.8.
The crashes are not specifically reproducible. Regardless of the load, the barebone (several HPE, Lenovo and Supermicro servers in use) or specific VMs (it needs always a group of VMs, doesn't matter which ones).
Some systems crashed several times a day, others only once a month. Some hosts had no crashes at all.
The only commonality: AMD Epyc and Genoa CPUs

We tried various configurations for ZFS, memory management, process management, power efficiency modes, KVM CPU models, with/without hyperthreading, etc. Nothing helped.

In the end, we took the 6.11 kernel from the Ubuntu repo (Proxmox is based on the Ubuntu kernel anyway) and since then the systems have been stable.
The 6.8 kernel definitely has a problem.

@proxmox Team
You should seriously consider retiring the 6.8 kernel and switching to 6.11.
Click to expand...
Thanks for sharing your experience! I would try it out, do you have any good instructions how to do it?
 
Just download the Kernel packages from an Ubuntu mirror (picked mirror.plusserver.com randomly):
It's ok for testing. For productive hosts you should think about own apt repos where you can store these packages. Or wait and hope that Proxmox will provide 6.11 kernels soon.

There is a package dependency for wireless-regdb. It's exists also in the Debian repos. Just install it before with apt.
Than install the deb packages directly.

You should load br_netfilter module explicitly:
Just add a line "br_netfilter" to "/etc/modules-load.d/br_netfilter.conf"

Because pve-firewall tries to read /proc/sys/net/bridge/bridge-nf-call-iptables which doesn't exists if the module isn't loaded.
Seems Proxmox patched this in their own version of the Ubuntu kernel. But the workaround is simple enough. :)

update-grub or the proxmox-boot-tool should set the 6.11 kernel as default because it's the highest number. So just reboot and hope :)

Of course you could also add the whole Ubuntu repo to your system. But that needs also some apt pinning configuration. Otherwise you probaly will fuck up your system if you start to mix all packages from Debian, Ubuntu and Proxmox. :p
 
  • Like
Reactions: Decco1337
HostingGuy said:
Just download the Kernel packages from an Ubuntu mirror (picked mirror.plusserver.com randomly):
It's ok for testing. For productive hosts you should think about own apt repos where you can store these packages. Or wait and hope that Proxmox will provide 6.11 kernels soon.

There is a package dependency for wireless-regdb. It's exists also in the Debian repos. Just install it before with apt.
Than install the deb packages directly.

You should load br_netfilter module explicitly:
Just add a line "br_netfilter" to "/etc/modules-load.d/br_netfilter.conf"

Because pve-firewall tries to read /proc/sys/net/bridge/bridge-nf-call-iptables which doesn't exists if the module isn't loaded.
Seems Proxmox patched this in their own version of the Ubuntu kernel. But the workaround is simple enough. :)

update-grub or the proxmox-boot-tool should set the 6.11 kernel as default because it's the highest number. So just reboot and hope :)

Of course you could also add the whole Ubuntu repo to your system. But that needs also some apt pinning configuration. Otherwise you probaly will fuck up your system if you start to mix all packages from Debian, Ubuntu and Proxmox. :p
Click to expand...
Thanks for that! I'll test it, I have enough servers which are empty to fuck them up haha :D Also need to test this for some days, I think it could be faster than waiting for Proxmox. I have issues since months with the 6.8 kernel and I have no patience anymore :D
 
HostingGuy said:
Just download the Kernel packages from an Ubuntu mirror (picked mirror.plusserver.com randomly):
It's ok for testing. For productive hosts you should think about own apt repos where you can store these packages. Or wait and hope that Proxmox will provide 6.11 kernels soon.

There is a package dependency for wireless-regdb. It's exists also in the Debian repos. Just install it before with apt.
Than install the deb packages directly.

You should load br_netfilter module explicitly:
Just add a line "br_netfilter" to "/etc/modules-load.d/br_netfilter.conf"

Because pve-firewall tries to read /proc/sys/net/bridge/bridge-nf-call-iptables which doesn't exists if the module isn't loaded.
Seems Proxmox patched this in their own version of the Ubuntu kernel. But the workaround is simple enough. :)

update-grub or the proxmox-boot-tool should set the 6.11 kernel as default because it's the highest number. So just reboot and hope :)

Of course you could also add the whole Ubuntu repo to your system. But that needs also some apt pinning configuration. Otherwise you probaly will fuck up your system if you start to mix all packages from Debian, Ubuntu and Proxmox. :p
Click to expand...
but zfs is not included, or?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back