Degraded performance with 5.15 kernel

[user18587]

We updated some of our debian 11 servers (proxmox 7) and it automatically pushed a kernel 5.15 upgrade (from 5.13).

After we rebooted we quickly noticed a huge difference in CPU usage as we have a pretty intensive workload on some servers, it became so much overloaded and unusable that we quickly tried to rollback to 5.13 which instantly fixed the problem.

We also tried to upgrade to 6.2, which worked fine, but this is not well maintained for proxmox 7 since last 6.2 kernel is from december so it doesn't include the fix for CVE-2024-1086 for example.

I can say that those servers that collapsed with 5.15 had AMD EPYC 7402P cpus , but i tested today and i think i noticed the same behavior on different hardware. I cannot say for sure that it's the CPU performance that degraded (or the consequence) but most likely.

Is there any hope to have another updated kernel (like 6.2) for proxmox 7 without having to install one from proxmox 8 manually?

Thanks

 

[shanreich]

You might have gotten hit with some mitigations for CPU-exploits, iirc several of them were introduced with 5.15 for AMD.

This can be particularly problematic when they are mitigated in the kernel, did you already check whether you have the newest microcode version available [1]? They can include mitigations that perform way better than when they are performed in the kernel.

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_firmware_cpu

 

[user18587]

That's true we don't have it installed, we will try it.

What is strange is that i can't reproduce it on 6.2.16-20 and i didn't read about a specific CPU vulnerability fix last few months

Thanks for your quick response

 

[shanreich]

That's true we don't have it installed, we will try it.

Yes, it might be best to do test on a test system, if possible or migrate your guests to other nodes if you have a cluster in order avoid potential disruptions in your services.

 

[fiona]

Hi,
please note that Proxmox VE 7 will be end-of-life with the end of July, so you should consider upgrading:
https://pve.proxmox.com/wiki/Upgrade_from_7_to_8