Fresh PVE install, no VMs running, all cores running (almost) at full Turboboost speed

Oct 21, 2024
21
1
3
I've installed Proxmox VE 8.4, no VMs running and noticed all cores are running at (almost) full speed. No ZFS or Ceph or other advanced features, just default-install with local-lvm.

Mainboard is Supermicro Mainboard X13SEI-TF, CPU is Intel Xeon Silver 4416+ (2,00 Ghz, 20-Core, 37,5MB).

I didn't change anything CPU-Scaling related in the BIOS. For testing I disabled Turboboost in the BIOS, this makes all cores run at 2000.000 when checking /proc/cpuinfo.

Anybody got an idea why that might be happening? Or any tips how to further narrow this down?



watch -n10 grep \'cpu MHz\' /proc/cpuinfo shows:
Code:
Every 10.0s: grep 'cpu MHz' /proc/cpuinfo

cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3886.173
cpu MHz         : 3774.579
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3900.000
cpu MHz         : 3891.719
cpu MHz         : 3796.537
cpu MHz         : 3900.000
cpu MHz         : 3900.000



top shows everything is idle:
Code:
top - 11:30:41 up 36 min,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 470 total,   1 running, 469 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 128353.7 total, 126234.5 free,   2814.0 used,    337.9 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used. 125539.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   7568 root      20   0   11680   5120   3200 R   0.3   0.0   0:00.01 top
      1 root      20   0  168216  11520   8960 S   0.0   0.0   0:01.67 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_release
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_g
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_p
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-slub_
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-netns
      9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri
     12 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-mm_pe
     13 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_kthread
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_rude_kthread
     15 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_trace_kthread
     16 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/0
     17 root      20   0       0      0      0 I   0.0   0.0   0:00.08 rcu_preempt
     18 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/0
     19 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/0
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     22 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/1
     23 root      rt   0       0      0      0 S   0.0   0.0   0:00.56 migration/1
     24 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/1
     25 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/1:0-events
     26 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/1:0H-events_highpri
     27 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2
     28 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/2
     29 root      rt   0       0      0      0 S   0.0   0.0   0:00.56 migration/2
     30 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/2
     31 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/2:0-events
     32 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/2:0H-events_highpri
     33 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3
     34 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/3
     35 root      rt   0       0      0      0 S   0.0   0.0   0:00.56 migration/3
     36 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/3
     38 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/3:0H-events_highpri
     39 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/4
     40 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/4
     41 root      rt   0       0      0      0 S   0.0   0.0   0:00.56 migration/4
     42 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/4
     43 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/4:0-mm_percpu_wq
     44 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/4:0H-events_highpri
     45 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/5
     46 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/5
     47 root      rt   0       0      0      0 S   0.0   0.0   0:00.56 migration/5
     48 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/5
     49 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/5:0-mm_percpu_wq
     50 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/5:0H-events_highpri
     51 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/6
     52 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/6
     53 root      rt   0       0      0      0 S   0.0   0.0   0:00.56 migration/6
     54 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/6
     56 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/6:0H-events_highpri
     57 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/7
     58 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/7
     59 root      rt   0       0      0      0 S   0.0   0.0   0:00.56 migration/7
     60 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/7
 
checked with "turbostat", interestingly, it gives different values compared to /proc/cpuinfo. Cores are all above 3GHz, but not (almost) all at 3900.

Code:
^CCore  CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     SMI     POLL    C1      C1E     C6      POLL%   C1%     C1E%    C6%     CPU%c1  CPU%c6  CoreTmp CoreThr PkgTmp  Pkg%pc2 Pkg%pc6 PkgWatt RAMWatt PKG_%   RAM_%   UncMHz
-       -       3       0.07    3680    2000    1.17    1900    0       89      105     931     1978    0.00    0.01    0.34    99.58   6.50    93.43   27      0       27      3.50    50.73   43.08   4.35    0.00    0.00    1400
0       0       8       0.20    3728    2000    0.99    407     0       0       1       348     122     0.00    0.00    3.96    95.86   14.86   84.93   25      0       27      3.50    50.73   43.09   4.35    0.00    0.00    1400
0       20      1       0.03    3127    2000    0.80    10      0       0       0       3       50      0.00    0.00    0.02    99.95   15.04
1       1       4       0.12    3845    2000    1.20    45      0       0       0       12      85      0.00    0.00    0.21    99.68   9.55    90.33   25      0
1       21      2       0.05    3718    2000    1.06    24      0       0       29      7       86      0.00    0.17    0.05    99.74   9.62
2       2       3       0.07    3769    2000    1.08    32      0       0       0       11      50      0.00    0.00    0.23    99.71   4.34    95.60   24      0
2       22      0       0.01    3320    2000    0.42    7       0       0       0       3       20      0.00    0.00    0.03    99.97   4.40
3       3       0       0.01    3312    2000    0.48    20      0       0       0       7       33      0.00    0.00    0.13    99.87   4.48    95.51   25      0
3       23      2       0.06    3805    2000    1.13    33      0       0       0       11      45      0.00    0.00    0.23    99.71   4.42
4       4       0       0.01    3290    2000    0.43    7       0       0       0       3       14      0.00    0.00    0.03    99.97   2.83    97.17   25      0
4       24      2       0.06    3782    2000    1.24    38      0       0       0       15      31      0.00    0.00    0.30    99.64   2.78
5       5       5       0.14    3684    2000    1.08    99      0       60      18      15      68      0.01    0.09    0.31    99.47   6.21    93.66   25      0
5       25      0       0.01    3296    2000    0.39    19      0       0       0       7       34      0.00    0.00    0.12    99.87   6.34
6       6       3       0.09    3660    2000    1.13    57      0       0       0       40      34      0.00    0.00    0.62    99.30   3.06    96.85   24      0
6       26      0       0.01    3173    2000    0.47    17      0       0       0       7       19      0.00    0.00    0.12    99.88   3.14
7       7       1       0.02    3429    2000    0.34    17      0       0       0       6       23      0.00    0.00    0.12    99.87   2.89    97.10   26      0
7       27      1       0.04    3458    2000    0.81    36      0       0       0       15      33      0.00    0.00    0.32    99.65   2.87
8       8       2       0.05    3484    2000    0.54    61      0       0       0       32      59      0.00    0.00    0.55    99.41   4.50    95.45   23      0
8       28      0       0.01    3208    2000    0.41    17      0       0       0       7       26      0.00    0.00    0.12    99.88   4.54
9       9       0       0.01    3234    2000    0.36    19      0       0       0       7       27      0.00    0.00    0.12    99.87   7.63    92.36   24      0
9       29      1       0.04    3448    2000    0.69    64      0       0       0       11      80      0.00    0.00    0.23    99.73   7.60
10      10      1       0.04    3465    2000    0.47    47      0       0       0       13      67      0.00    0.00    0.26    99.71   7.70    92.27   24      0
10      30      0       0.01    3481    2000    0.30    20      0       0       0       7       46      0.00    0.00    0.12    99.87   7.72
11      11      4       0.11    3498    2000    0.90    79      0       0       0       27      107     0.00    0.00    0.50    99.39   10.88   89.00   27      0
11      31      0       0.01    3312    2000    0.37    24      0       0       0       10      55      0.00    0.00    0.13    99.86   10.98
12      12      6       0.17    3716    2000    0.89    54      0       0       4       13      101     0.00    0.10    0.24    99.50   13.73   86.11   24      0
12      32      2       0.06    3500    2000    0.36    77      0       29      49      9       106     0.00    0.16    0.17    99.61   13.83
13      13      0       0.01    3200    2000    0.39    25      0       0       1       11      29      0.00    0.00    0.22    99.78   5.25    94.74   23      0
13      33      14      0.36    3824    2000    1.56    70      0       0       0       61      41      0.00    0.00    0.80    98.84   4.90
14      14      9       0.25    3651    2000    1.41    97      0       0       0       40      79      0.00    0.00    0.75    99.01   7.52    92.23   25      0
14      34      0       0.01    3153    2000    0.38    17      0       0       0       7       37      0.00    0.00    0.12    99.87   7.76
15      15      10      0.25    3842    2000    1.52    70      0       0       0       32      49      0.00    0.00    0.49    99.27   5.07    94.68   25      0
15      35      0       0.01    3013    2000    0.41    21      0       0       0       11      28      0.00    0.00    0.22    99.78   5.31
16      16      1       0.01    3432    2000    0.34    16      0       0       1       2       30      0.00    0.00    0.03    99.96   4.11    95.87   25      0
16      36      0       0.01    3168    2000    0.42    21      0       0       2       7       29      0.00    0.00    0.13    99.87   4.12
17      17      0       0.01    2973    2000    0.43    28      0       0       0       12      32      0.00    0.00    0.16    99.83   4.16    95.83   26      0
17      37      1       0.02    3465    2000    0.31    34      0       0       0       7       42      0.00    0.00    0.10    99.89   4.15
18      18      0       0.01    3128    2000    0.37    21      0       0       0       6       29      0.00    0.00    0.12    99.87   3.61    96.38   24      0
18      38      0       0.01    3188    2000    0.36    21      0       0       0       7       29      0.00    0.00    0.11    99.88   3.61
19      19      13      0.35    3772    2000    1.40    103     0       0       0       74      63      0.00    0.00    0.99    98.67   7.15    92.50   25      0
19      39      2       0.06    3445    2000    2.00    26      0       0       0       8       40      0.00    0.00    0.12    99.83   7.44
 
Found out that for whatever (performance?) reason, Proxmox defaults to the performance governor, not ondemand or powersave.

Still not sure why /proc/cpuinfo and turbostat report different values. Whatever, everything's running fine so far.
 
Hi,
Would you mind sharing how you installed turbostat on proxmox ?

Seems like it should be part of package linux-tools

maybe
Code:
apt install linux-tools-common linux-tools-$(uname -r)

maybe perf-tools

but none appear to exist

Code:
E: Package 'linux-tools-common' has no installation candidate
E: Unable to locate package linux-tools
E: Unable to locate package linux-tools-generic
E: Unable to locate package perf-tools
 
Thanks ! even with chatgpt and googling, I just couldn't find it !

I've been having reliably intermittent instability crashing my server and I'm narrowing it down to the c-states
I've been investigating a lot about the system, dmidecode and sensors and that kind of stuff, I'm surprised that Proxmox doesn't have health monitoring out of the box, there is so much stuff to add !