Dell R520 Recurring Problem CPU locking

[Mihai]

I've been working with a Dell R520 with dual Xeon(R) CPU E5-2430 v2 @ 2.50GHz and I've had this recurring issue (for multiple years) after various kernel updates where the virtual machines (both windows and linux) lock up. Previously, I was somehow able to rectify this issue by disabling as many CPU features as possible in the BIOS, but this is no longer possible. I would say this has happened about 3 times previously.

This happened with Kernels 5.15 and I upgraded to kernel 5.19 thinking it might solve it, but it has not.

All hardware tests are passing and everything seems to be working fine as far as I can tell, except the virtual machines are not able to run. Ceph is working correctly and the cluster is fine, except that I can't run any VMs on this node.

I don't see any errors show up in the Syslog.

The cluster is 3-notes and the backend is a CEPH cluster.

My only solution is to replace the whole server.

After a machine has been running for 5 or so minutes, past the grub loader, the following can be seen on the console:



Does anyone know what the issue can be or how I can diagnose it?

 

[spirit]

I have r520 in production with same kind of processors, I never had seen this error.

Is your bios updated ? Do you have installed intel-microcode package to have recent cpu update ?

Do you have any error in your idrac lifecycle logs ?

 

[Mihai]

I have r520 in production with same kind of processors, I never had seen this error.

Is your bios updated ? Do you have installed intel-microcode package to have recent cpu update ?

Do you have any error in your idrac lifecycle logs ?

I've got BIOS Version 2.9.0.

The only critical Licecycle Errors in the past 2 years are chassis opened, drives removed and power supply redundancy lost (regular stuff). Warnings have to do with NIC ports going up and down.

I've never installed any intel-microcode package for a CPU update. I'm willing to give it a try. Do you recommend this method?

 

[Mihai]

I installed the intel-microcode package and restarted then I used all of the following commands to check for a microcode update and there were none.

dmesg | grep "microcode updated early to"
journalctl -b -k | grep "microcode updated early to"
zgrep "microcode updated early to" /var/log/kern.log*

I checked /proc/cpuinfo for the microcode and here's what I get:

microcode       : 0x42e

Looking at this microcode update guide under Ivy bridge Xeon Procesor E5 v2, it says the production MCU Rev is 0x42E, which is what I already have.

I still have the same issue =(

 

[spirit]

can you try to add in /etc/default/grub

GRUB_CMDLINE_LINUX="idle=poll intel_idle.max_cstate=0 intel_pstate=disable processor.max_cstate=1"

then "# update-grub"

and reboot ?


This should force cpu to never sleep, maybe it could help.

 

[Mihai]

can you try to add in /etc/default/grub

GRUB_CMDLINE_LINUX="idle=poll intel_idle.max_cstate=0 intel_pstate=disable processor.max_cstate=1"

then "# update-grub"

and reboot ?


This should force cpu to never sleep, maybe it could help.

Thanks for the tip but unfortunately, it did not solve the issue, and it also made the server fans run at 100% so I reverted.