[SOLVED] High load on idle system

[teegee]

Hi,
I'm observing high system load on a proxmox VE where nothing is happening and I can't find the source of it.
I set this server up a few weeks ago with only 4 lxc containers and no vms.
lxc are set to NOT start on boot but after a reboot system load immediately jumps to 4-5 and stays there.

The system is runing pve-manager 8.2.4 fully dist-upgraded.
CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
RAM: 64GB

top - 13:51:50 up 3 days, 15:55, 1 user, load average: 4.17, 4.62, 4.61
Tasks: 354 total, 1 running, 353 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.3 sy, 0.0 ni, 99.4 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 64237.1 total, 49629.6 free, 2470.7 used, 12936.6 buff/cache
MiB Swap: 8192.0 total, 8192.0 free, 0.0 used. 61766.5 avail Mem

There is no significant disk IO or network IO.

# ps -eo state,pid,cmd | grep "^L"
# ps -eo state,pid,cmd | grep "^Z"
# ps -eo state,pid,cmd | grep "^W"
# ps -eo state,pid,cmd | grep "^T"
[ all show nothing ]
# ps -eo state,pid,cmd | grep "^D"
D     195 [kworker/7:1H+acpi_thermal_pm]
D 1277390 [kworker/7:1+kacpi_notify]
D 1287455 [kworker/1:2+kacpi_notify]
D 1292067 [kworker/u24:1+events_freezable_power_]
D 1292149 [kworker/7:3+kacpi_notify]
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 50835412 317728 12930164    0    0    16     4   22   23  0  0 100  0  0
 1  0      0 50835412 317728 12930164    0    0   768     0 1297 1028  0  1 99  0  0
 1  0      0 50835412 317728 12930164    0    0     0    12  234  254  0  0 100  0  0
 1  0      0 50835412 317728 12930164    0    0     0     4  305  212  0  0 100  0  0
 1  0      0 50835412 317728 12930164    0    0     0    40  206  228  0  0 100  0  0
 1  0      0 50835412 317728 12930164    0    0     0    32  136  183  0  0 100  0  0
 1  0      0 50835412 317728 12930164    0    0     0     0  191  247  0  0 100  0  0

Any ideas what's going on?

 

[LnxBil]

Can you try to disable all power saving features in the BIOS?

 

[teegee]

I made the following BIOS changes:
- Disable Intel Speed Step (EIST)
- Disable C3, C6/C7, C8, C10 CPU C-States
- Disable CPU enhanced halt (C1E)
- Set Package C State Limit to C0/C1
- Disable Hardware Prefetcher
- Disable Adjacent Cache Line Prefetch
- Disable Voltage Optimisation
- Disable Energy Efficient Turbo
- Disable Intel Speed Shift Technology

After rebooting, load settled around 2.8.

Then I went to check the BIOS version and noticed there are updates available. Installed the update and system load dropped to near zero with no containers running.
After starting all the containers, load is still near zero.

Turns out, if you have unexplained issues, updating the BIOS can do the trick.
Cheers