High CPU usage after upgrade to kernel 6.5.13-1-pve

Polyphemus

Member
Nov 18, 2021
45
9
13
43
Hi,

This morning I upgraded my server to kernel 6.5.13-1-pve. Many of my containers and VMs now have a very high CPU usage. Overall, my Ryzen 5 5600G had a usage of around 20% average before the upgrade, now it hovers around 70%.

CPU.jpg

Code:
proxmox-ve: 8.1.0 (running kernel: 6.5.13-1-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
pve-kernel-5.15: 7.4-8
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-5.15.131-1-pve: 5.15.131-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.4
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1

How can I debug?
 
Last edited:
Hi,
how did the RAM usage look like during the time of the issue? What kind of workload is running in the guests? Should you be able to boot into the new kernel again, can you check what processes/threads in particular are responsible for the CPU usage, e.g. using htop?
 
  • Like
Reactions: Polyphemus
Hi,
how did the RAM usage look like during the time of the issue? What kind of workload is running in the guests? Should you be able to boot into the new kernel again, can you check what processes/threads in particular are responsible for the CPU usage, e.g. using htop?
Hi,

RAM usage was the same at the time of the issue, of slightly elevated (20GB against 18GB). If I recall correctly it were Node.js (and some Python and Go) processes in a variety of LXCs.

I can't debug right now, as my home network uses one of the LXCs as DNS service. I hope I can try debugging it tonight.
 
Hi Chris, hi Fiona,

Just updated to the latest BIOS version (December 25, 2023), microcode is up to date, I rebooted into 6.5.13-1, and still all CPU cores (12) have a 100% usage. There aren't any specific processes causing the high load, as they are all competing each other as it seems with high usage. It's like having a single core CPU, even though all cores are shown in htop.

Rebooted back into 6.5.11-8 and everything is fine...
 
Well, I tried something:

On 6.5.13-1, running
Code:
cpufreq-info frequency-info
all cores run at a current (fixed?) CPU frequency of 399/400 MHz:

Code:
root@pve-r5:~# cpufreq-info freqency-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 399 MHz.
analyzing CPU 1:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 1
  CPUs which need to have their frequency coordinated by software: 1
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 399 MHz.
analyzing CPU 2:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 2
  CPUs which need to have their frequency coordinated by software: 2
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 399 MHz.
analyzing CPU 3:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 3
  CPUs which need to have their frequency coordinated by software: 3
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
analyzing CPU 4:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 4
  CPUs which need to have their frequency coordinated by software: 4
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 399 MHz.
analyzing CPU 5:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 5
  CPUs which need to have their frequency coordinated by software: 5
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
analyzing CPU 6:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 6
  CPUs which need to have their frequency coordinated by software: 6
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
analyzing CPU 7:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 7
  CPUs which need to have their frequency coordinated by software: 7
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 399 MHz.
analyzing CPU 8:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 8
  CPUs which need to have their frequency coordinated by software: 8
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
analyzing CPU 9:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 9
  CPUs which need to have their frequency coordinated by software: 9
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
analyzing CPU 10:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 10
  CPUs which need to have their frequency coordinated by software: 10
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 399 MHz.
analyzing CPU 11:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 11
  CPUs which need to have their frequency coordinated by software: 11
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.

On 6.5.11-8, running the same command:

Code:
root@pve-r5:~# cpufreq-info freqency-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 4.44 GHz.
analyzing CPU 1:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 1
  CPUs which need to have their frequency coordinated by software: 1
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 3.57 GHz.
analyzing CPU 2:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 2
  CPUs which need to have their frequency coordinated by software: 2
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 3.53 GHz.
analyzing CPU 3:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 3
  CPUs which need to have their frequency coordinated by software: 3
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
analyzing CPU 4:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 4
  CPUs which need to have their frequency coordinated by software: 4
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 3.55 GHz.
analyzing CPU 5:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 5
  CPUs which need to have their frequency coordinated by software: 5
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
analyzing CPU 6:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 6
  CPUs which need to have their frequency coordinated by software: 6
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
analyzing CPU 7:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 7
  CPUs which need to have their frequency coordinated by software: 7
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 3.57 GHz.
analyzing CPU 8:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 8
  CPUs which need to have their frequency coordinated by software: 8
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 4.13 GHz.
analyzing CPU 9:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 9
  CPUs which need to have their frequency coordinated by software: 9
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 3.89 GHz.
analyzing CPU 10:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 10
  CPUs which need to have their frequency coordinated by software: 10
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 3.55 GHz.
analyzing CPU 11:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 11
  CPUs which need to have their frequency coordinated by software: 11
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency is 4.02 GHz.

Cores are boosting to 4+ GHz.
 
Last edited:
Can you check if the CPU usage gets lower with kernel 6.5.13 when you switch to the performance governor?
 
Sadly, that doesn't change anything:

Code:
analyzing CPU 11:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 11
  CPUs which need to have their frequency coordinated by software: 11
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
 
Sadly, that doesn't change anything:

Code:
analyzing CPU 11:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 11
  CPUs which need to have their frequency coordinated by software: 11
  maximum transition latency: 4294.55 ms.
  hardware limits: 400 MHz - 4.46 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 400 MHz and 4.46 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 400 MHz.
Did you also check the actual usage or just the frequency value?
 
Actual usage is 100%, I just only copied CPU core 11, for the readability. All cores show current CPU frequency of 400 MHz.

Code:
root@pve-r5:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.13-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt acpi_enforce_resources=lax

I've added
Code:
amd_pstate=guided
. Will check and report back.
 
I now see the cpu driver has changed from amd-pstate-epp to amd-pstate and using performance as governor boosts all cores to thier maximum. Maximum latency dropped to 20 us, from 4500 ms.

Using powersave leaves al cores fixed at 400 MHz.

The reason I'm using powersave is that I'm trying to make my server more power efficient, if that's even possible with a Ryzen. Now, with the current config (amd-pstate and using performance), I've got a hunch it runs at maximum performance/power usage.

On 6.5.11-8 powersave was scaling, now on 6.5.13-1 it doesn't, if I may think out loud.
 
Last edited:
I now see the cpu driver has changed from amd-pstate-epp to amd-pstate and using performance as governor boosts all cores to thier maximum. Maximum latency dropped to 20 us, from 4500 ms.

Using powersave leaves al cores fixed at 400 MHz.

The reason I'm using powersave is that I'm trying to make my server more power efficient, if that's even possible with a Ryzen. Now, with the current config (amd-pstate and using performance), I've got a hunch it runs at maximum performance/power usage.
Did the CPU usage change or is it still at 100%? With both performance or powersave? Is there now an ondemand governor available as well?
 
Did the CPU usage change or is it still at 100%? With both performance or powersave? Is there now an ondemand governor available as well?
When using powersave all cores use 100% persistently, when using performance all cores are using 5% to 15% and some lower as the processes jump around the cores.

The following governors are available:
available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil

To add, I'm using the cpufrequtils service to change the governor.
 
Try setting the amd_pstate=passive in the kernel command line, this should not enable the firmware related amd-pstate-epp driver instead of active or guided. Then you might try setting a different governor for frequency scaling, e.g. powersave.

See also https://www.kernel.org/doc/html/latest/admin-guide/pm/amd-pstate.html#active-mode

On a side note: You do not require the intel related kernel parameters, please remove them.
 
Changed to amd_pstate=passive and when using powersave, still fixes the cores at 400MHz, with the driver: amd-pstate. amd_pstate=guided uses the amd-pstate driver too, and not the amd-pstate-epp
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!