Lenovo SR550 High Fan Speed (100%)

[oodissimo]

Hi,

Running Proxmox 8.0 or 8.0 upgraded to latest using the no-subscription repo on a Lenovo SR550. From the moment Prox max gets past the grub stage, the fans ramp up to 100% and stay at 100%.

This seems to be unique to running Proxmox. When running RHEL9, Ubuntu 20.04 or Debian Bookworm this does not happen. I am at a bit of a loss where to even begin troubleshooting this. Running `htop` CPU usage is 0% across all 40 threads. Following some other threads, I looked for `/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`. Unfortunately I have to say there are no such directories, there is no cpu* (or cpu0, cpu1, cpu2, etc) in `sys/devices/system/cpu/`.

I am running other systems with Proxmox such as, NUC10, HP Prodesk SFF, Dell 12th gen servers and Dell 13th gen servers. None of those exhibit the high fan usage. The two other systems I checked do have `/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`.

Any suggestions on where to start looking?

 

[zzz09700]

Usually this is due to some ACPI stuff in Linux not seeing a corporating MB.

Windows and ESXi handles broken ACPI stuff better than Linux kernel so you may want to try those.

Alternativly, you can do some custom wiring and snap in a stand alone PWM fan controller, which has its own MCU, fan curve and temp sensors. I did this to several Supermicro MB as they refused to corporate with workstation fans (~3000RPM max and usually runs at ~1000RPM)

 

[gfngfn256]

Debian Bookworm this does not happen

So when you run Debian or Ubuntu on this Lenovo SR550, do you have something in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
If you do. Then it would be very weird that under Proxmox you don't - as Proxmox is basically Debian under the hood with a modified Ubuntu kernel.

I don't know the install history of your Lenovo SR550 - but I suggest a re-install.

 

[oodissimo]

So when you run Debian or Ubuntu on this Lenovo SR550, do you have something in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
If you do. Then it would be very weird that under Proxmox you don't - as Proxmox is basically Debian under the hood with a modified Ubuntu kernel.

I don't know the install history of your Lenovo SR550 - but I suggest a re-install.

Yes, trying RHEL9 (already installed on an SDD and used for all of the hardware firmware upgrades) next. Then doing a Debian Bookworm install to see what it brings. I have had Debian Bookworm up and running previously, but Lenovo firmware updates demand a genuine RHEL9 base to run their tools from.
The Proxmox install is brand new, first using the 8.2-1 installer, where there the Proxmox install won't get past loading initrd. Then the 8.0 installer which did provide a bootable system. There are other threads on the "hanging at initrd" issue.

 

[gfngfn256]

The Proxmox install is brand new, first using the 8.2-1 installer, where there the Proxmox install won't get past loading initrd. Then the 8.0 installer which did provide a bootable system. There are other threads on the "hanging at initrd" issue.

OK, I see your post here on your install. What I would do IIWY try a fresh Debian install & if all works (bootup & fans), install Proxmox on top. This may or may not work. As if its a direct kernel problem, you'll end up in the same boat. See here on Proxmox VE on Debian 12.

 

[oodissimo]

OK, I see your post here on your install. What I would do IIWY try a fresh Debian install & if all works (bootup & fans), install Proxmox on top. This may or may not work. As if its a direct kernel problem, you'll end up in the same boat. See here on Proxmox VE on Debian 12.

Mystery has been solved, somewhat. I am working with several nearly identical servers. Nearly being the key word here. They all have PCIe cards with an NVMe as a boot disk. One of them had a Samsung 960PRO. It is the Samsung NVme that is causing the high fan speed. The same occurs with a Samsung 970. It does not happen with a HP EX900 NVMe nor Western Digital NVMe.

 

[zzz09700]

That's rather strange. If BMC decides to go full blow because it sees a unsupported or buggy PCIe device then it would do so from POST, not after grub starts loading OS.

 

[oodissimo]

That's rather strange. If BMC decides to go full blow because it sees a unsupported or buggy PCIe device then it would do so from POST, not after grub starts loading OS.

I am using a Samsung 960PRO NVMe in a PCIe card as boot disk in a 13th gen Dell. No issues there, doesn't trigger the third party fan response either. Also using Samsung 970 EVO as data disks in 2 NUC10. Neither seeing any issues there. Seems to be specific to the Samsung NVMe in the Lenovo SR550. I have different types of PCIe cards with M.s slots, I doesn't matter which on of them I try, makes no difference. Samsung NVMe high fan speed, HP or WD NVMe all is normal.

 

[gfngfn256]

Seems to be specific to the Samsung NVMe in the Lenovo SR550

Update the firmware of the NVMe?

 

[zzz09700]

Just some background knowledge, Samsung has always been a major offender for many industrial standards inside a server case.

We ditched Samsung back in the days of SATA SSDs, life was much better after that.

 

[zzz09700]

Update the firmware of the NVMe?

Don't do it.
With Samsung's firmware update tools you are more likely to brick your SSD. We have seen this enough times from multiple unsuspecting clients.

Stay with Intel for your enterprise SSDs and Micron/WD for consumer SSDs. They may not have the best performance but they do comply with existing industrial stands, in a fashion Samung cannot even begin to understand. (Or they do understand, just too arrogant to give a shit)

Heck even cheap Chinese SSDs do better than Samsung in this respect.

 

[oodissimo]

Don't do it.
With Samsung's firmware update tools you are more likely to brick your SSD. We have seen this enough times from multiple unsuspecting clients.

Stay with Intel for your enterprise SSDs and Micron/WD for consumer SSDs. They may not have the best performance but they do comply with existing industrial stands, in a fashion Samung cannot even begin to understand. (Or they do understand, just too arrogant to give a shit)

Heck even cheap Chinese SSDs do better than Samsung in this respect.

Whenever I checked specs, the TBW of Samsung far exceeded that of WD. Speed is of little concern in the hardware available to me. Not finding any noticeable difference in SATA SSD versus NVMe speeds from a usage perspective. Not doing benchmarking just the non-scientific observation.

Choosing NVMe has been driven by being able to add yet another drive using a PCIe slot and not needing a SATA/SAS port.