Performance drop-off after upgrade from Proxmox v8.4.19 to v9.2.1

I

iay

Member
Dec 16, 2023
13
11
8
I have a small hobbyist Proxmox cluster with four nodes, which has been running Proxmox v8. I do not have a subscription. I noticed that this has started flagging the end of support for v8 and so today I migrated the least important (smallest, slowest) of the nodes to v9.

The node in question is an old, slow one (2-core Penryn, 8GB of RAM) which has been running three VMs (one Debian 11, one Red Hat 8, one Rocky 8) for years. Historically the CPU on this node has been about 20% on the host side. There's enough RAM for everything; these are very small machines with 1GB min / 2GB max memory set. Each machine is set to use 2 vCPU cores. The overallocation has never been a problem in the past.

After upgrading to V9.2.4 today, the CPU on this node is essentially permanently pegged at 100%, load average of about 7, with "top" showing the "kvm" processes for the three VMs being essentially all of this. In the GUI, it's a solid red bar labelled "100.00% of 2 CPUs".

It looks like running the first of the three VMs behaves normally, but the host CPU percentage of both jumps up as soon as I start the second one. I haven't run this in all combinations yet.

The storage for all three VMs is over NFS to a NAS, but that is also unchanged so I don't think that's the root cause here.

I have tried rebooting the host, and each of the VMs, without any change in behaviour. I was using HA on the cluster, mainly for maintenance mode migrations, but I have removed the VMs on this node from HA now just in case and again that hasn't changed anything.

Does anyone have any clues as to what might be going on here, or how I might start to figure it out?

I suppose it's possible that part of the problem is that I am running a cluster with one v9 node and three v8 ones, but by the same token this has so completely sandbagged the first node that I am afraid that if I upgrade the others that this might happen there too, which would be bad.
 
Please share the output of
Bash: 
top -coCPU%
lscpu 

# For each that has elevated utilization
qm config VMID

apt -yU install linux-perf
perf top --sort pid,dso,symbol
 
Thanks for the suggestions. I think I can see a couple of possibilities.

Code: 
(start of top -co\%CPU)
2814 root      20   0 3486728   1.3g  21432 R  75.8  17.2      6,08 /usr/bin/kvm -id 206 -n+
1545 root      20   0 3595184 992308  21168 S  75.5  12.6 295:01.34 /usr/bin/kvm -id 204 -n+
3930 root      20   0 4091204   2.0g  21380 S  44.0  26.0 316:39.23 /usr/bin/kvm -id 207 -n+

Code: 
lscpu:
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             36 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      2
  On-line CPU(s) list:       0,1
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Core(TM)2 Duo CPU     P8800  @ 2.66GHz
    CPU family:              6
    Model:                   23
    Thread(s) per core:      1
    Core(s) per socket:      2
    Socket(s):               1
    Stepping:                10
    CPU(s) scaling MHz:      100%
    CPU max MHz:             2660.0000
    CPU min MHz:             1596.0000
    BogoMIPS:                5309.19
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat p
                             se36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm con
                             stant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni
                             dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 x
                             save lahf_lm pti tpr_shadow flexpriority vpid dtherm vnmi
Virtualization features:     
  Virtualization:            VT-x
Caches (sum of all):         
  L1d:                       64 KiB (2 instances)
  L1i:                       64 KiB (2 instances)
  L2:                        3 MiB (1 instance)
NUMA:                       
  NUMA node(s):              1
  NUMA node0 CPU(s):         0,1
Vulnerabilities:             
  Gather data sampling:      Not affected
  Ghostwrite:                Not affected
  Indirect target selection: Not affected
  Itlb multihit:             KVM: Mitigation: Split huge pages
  L1tf:                      Mitigation; PTE Inversion; VMX EPT disabled
  Mds:                       Vulnerable: Clear CPU buffers attempted, no microcode; SMT disabled
  Meltdown:                  Mitigation; PTI
  Mmio stale data:           Not affected
  Old microcode:             Not affected
  Reg file data sampling:    Not affected
  Retbleed:                  Not affected
  Spec rstack overflow:      Not affected
  Spec store bypass:         Vulnerable
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitizatio
                             n
  Spectre v2:                Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS No
                             t affected; BHI Not affected
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Not affected
  Vmscape:                   Not affected

Code: 
qm config 204
agent: 1
balloon: 1024
boot: order=scsi0
cores: 2
cpu: Penryn
memory: 2048
name: d11a
net0: vmxnet3=00:50:56:bb:00:78,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
protection: 1
scsi0: pvelib04n:204/vm-204-disk-0.qcow2,discard=on,iothread=1,size=20G
scsihw: virtio-scsi-single
smbios1: uuid=efe1d51c-1173-4739-abdc-e6d8acafb5e8
sockets: 1
tags: ha;01;penryn;tester
vmgenid: 702d6349-81d9-43b5-b978-28aa5e0fb6fc


Code: 
qm config 206
agent: 1
balloon: 1024
boot: order=scsi0
cores: 2
cpu: Penryn
description: # `rl801`%0A%0ARocky Linux 8 testing.
memory: 2048
name: rl801
net0: vmxnet3=00:0C:29:D6:BC:8B,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
protection: 1
scsi0: pvelib04n:206/vm-206-disk-0.qcow2,discard=on,iothread=1,size=16G
scsihw: virtio-scsi-single
smbios1: uuid=e668d697-8134-40b5-a943-1c2a0e2c2a1a
sockets: 1
tags: ha;01;penryn;tester
vmgenid: bf9d9ceb-b801-4f60-bae2-a39bbf66e0de

Code: 
qm config 207
agent: 1
balloon: 1024
bios: ovmf
boot: order=scsi0
cores: 2
cpu: Penryn
description: # `r801`%0A%0ARed Hat Enterprise Linux 8 testing.%0A%0A*Note%3A this machine uses EFI and secure boot.*
efidisk0: pvelib04n:207/vm-207-disk-0.qcow2,efitype=4m,ms-cert=2023k,pre-enrolled-keys=1,size=528K
machine: q35
memory: 2560
name: r801
net0: vmxnet3=00:0C:29:1B:33:B9,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
protection: 1
scsi0: pvelib04n:207/vm-207-disk-1.qcow2,discard=on,iothread=1,size=20G
scsihw: virtio-scsi-single
smbios1: uuid=1adea127-f642-4373-b004-5dd441373eb1
sockets: 1
tags: ha;01;penryn;tester
vmgenid: cc1ed70c-52c5-4cc0-828c-42f76dcb8879

Code: 
perf top --sort pid,dso,symbol
Samples: 38K of event 'cycles:P', 4000 Hz, Event count (approx.): 8775843562 los
Overhead      Pid:Command          Shared Object            Symbol
  13.13%     2814:kvm              [kernel]                 [k] read_hpet
   4.03%        0:swapper          [kernel]                 [k] read_hpet
   1.42%     2814:kvm              [kernel]                 [k] entry_SYSCALL_64
   1.17%        0:swapper          [kernel]                 [k] hpet_clkevt_set_
   1.07%     2814:kvm              [vdso]                   [.] __vdso_clock_get
   0.91%     2814:kvm              [kernel]                 [k] syscall_return_v
   0.89%        0:swapper          [kernel]                 [k] cpu_idle_poll.is
   0.79%   163977:perf             [kernel]                 [k] vsnprintf
   0.79%   163977:perf             [kernel]                 [k] module_get_kalls
   0.74%     1005:corosync         [kernel]                 [k] read_hpet
   0.73%   163977:perf             [kernel]                 [k] kallsyms_expand_
   0.69%     2814:kvm              [kernel]                 [k] entry_SYSRETQ_un
   0.64%   163977:perf             [kernel]                 [k] format_decode
   0.64%     2814:kvm              [kernel]                 [k] entry_SYSCALL_64
   0.58%     1034:pvestatd         [kernel]                 [k] copy_page_range
   0.57%        0:swapper          [kernel]                 [k] tick_check_broad
   0.54%     1601:CPU 0/KVM        [kernel]                 [k] read_hpet
   0.51%     3930:kvm              [kernel]                 [k] read_hpet
   0.49%   163977:perf             [kernel]                 [k] string

I also ran `mpstat` and got results like this, confirming (I think) that we're in the kernel most of the time:

Code: 
Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all    5.16    0.00   85.70    0.00    0.00    0.40    0.00    8.64    0.00    0.10
Average:       0    7.75    0.00   81.91    0.00    0.00    0.40    0.00    9.94    0.00    0.00
Average:       1    2.58    0.00   89.48    0.00    0.00    0.40    0.00    7.34    0.00    0.20

One thing I noticed that may not be relevant at all is that all three VMs running on this machine were originally imported from ESXi, so they have `vmxnet3` NICs. I suppose it's possible that the issue stems from that code. Most of the machines on other hosts are using `virtio` devices and it's probably reasonable to update these as well (I did so for the SCSI controllers a while back).

The other thing was that `read_hpet` seems to imply we're spending a lot of time figuring out what the time is. Some searching indicates that perhaps this is the slowest available form of timer. Here's some possibly relevant output:

Code: 
root@pve01:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
hpet acpi_pm
root@pve01:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet
root@pve01:~# dmesg | grep -E "clocksource|tsc|hpet"
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2654.598 MHz processor
[    0.014974] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.281821] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417870 ns
[    0.286842] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2643b4db181, max_idle_ns: 440795271314 ns
[    0.464959] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.550114] tsc: Marking TSC unstable due to TSC halts in idle
[    0.550114] hpet: 4 channels of 0 reserved for per-cpu timers
[    0.550114] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31, 31
[    0.550114] hpet0: 4 comparators, 64-bit 25.000000 MHz counter
[    0.562466] clocksource: Switched to clocksource hpet
[    0.573066] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns

On the other machines I seem to be using the `tsc` clock source, but they are all running a 6.8.12 kernel and not the 7.0.14 that the upgraded machine is running. So this could be a change in the way the kernel determines which clock source to use on this (ancient and terrible) hardware.

I found a Red Hat article saying that "The HPET clocksource is considerably slower to access than the TSC, leading to increased CPU overhead in the read_hpet() function" which would track.

I suppose I can try booting under the old kernel to see whether there's something informative in `dmesg` there on this hardware, and indeed whether I ever was using TSC there. I'm guessing actually running PVE 9 over a 6.8.12 kernel long term is probably not a great idea even if it works, but I think v9 was released with something older than 7.x so perhaps it might work as a stopgap and as a debug tool.
 
  • Like
Reactions: Onslow
I did a "proxmox-boot-tool kernel pin 6.8.12-32-pve --next-boot" to boot the old kernel. The available and current clock sources end up the same, but with slightly different logging:

Code: 
dmesg | grep -E "clocksource|tsc|hpet"
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2654.831 MHz processor
[    0.073363] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.277992] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417870 ns
[    0.283012] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2644911299d, max_idle_ns: 440795264259 ns
[    0.458700] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.542045] hpet: 4 channels of 0 reserved for per-cpu timers
[    0.542050] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31, 31
[    0.542057] hpet0: 4 comparators, 64-bit 25.000000 MHz counter
[    0.544059] clocksource: Switched to clocksource tsc-early
[    0.554442] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.610918] tsc: Marking TSC unstable due to TSC halts in idle
[    0.611664] clocksource: Switched to clocksource hpet

The VMs do run in this configuration, and the 100% CPU utilisation is not there. I do notice a very high I/O delay statistic, which is peculiar because these machines aren't doing any I/O (if you don't count avahi-daemon and the SSH sessions running "top"). There is zero disk I/O.

I scrolled down a bit in the new v9 user interface and noticed some new graphs for CPU pressure stall and IO pressure stall.

On the new kernel, CPU pressure stall on all three machines is "some" at 50%+, "full" at 25% or so. On the old kernel, this is around 5%.

On the new kernel, IO pressure stall is essentially zero (but maybe that doesn't count because the other kind of stalling was dominant); on the old kernel, it's around 80% on all three machines.

I am none the wiser for this. The system certainly seems to run much better under the old kernel, though, and it doesn't seem to be due to a change in clock source determination.
 
Last edited:
It turns out that I have a couple of weeks of Prometheus metrics for this node. Both the CPU pressure stalls and I/O pressure stalls started at the upgrade to PVE 9 and were not present in PVE 8.

The CPU pressure stalls seem to be related to the kernel, as they go away when I revert that.

The IO pressure stalls don't go away when the kernel is reverted, so I guess they might be a QEMU issue (I guess there's not much code that can affect this other than the kernel inside the VM, QEMU and the kernel on the host). I see a couple of other reports of this but the favoured solution seems to be to remove external USB storage as in JetKVMs and this is a completely bare machine, just a physical monitor and keyboard plus network plugged in right now.

There are some forum threads indicating that there was a cosmetic problem like this with pve-qemu-kvm 10.2.1-1, but that they were fixed. My system is running 11.0.2-1 so is presumably after that. In addition, there is no disk I/O at all so it's hard to see this coming from the aio setting (which is defaulted to io_uring in these machines). I will try some tweaks, though.
 
Last edited:
Disclaimer, I'm not very familiar with how these changes perform but you can try if it behaves different with these kernel args
Bash: 
clocksource=tsc tsc=reliable
Other things you can test are changing the NIC to virtio and CPU to host.
 
Last edited:
I tried setting the NIC to virtio and the CPU to host (it was already set to Penryn, which is what the host is; I set it explicitly so migration would work). No change either under the old kernel (CPU performance is still fine but IO wait is excessive even without disk I/O happening) or the new kernel (CPU pegs at 100%).

I might try setting the clock source in the kernel args, although I'm a bit concerned it might be unstable. Unfortunately the first thing I see on the screen when the machine boots is the LVM activation, I never see the grub screen so I don't have the opportunity to set it manually. I can't remember if it was always this way or if I had the screen plugged in differently before and my notes don't say. I also can't remember how to set those in grub options, but I will look into it. Thanks again.
 
Not being able to get in to the grub menu turned out to be down to grub being in some kind of EFI graphical mode that doesn't work on this hardware. Avoided with the nomodeset option and enabling the console, so now I can add things at boot time like I remember.

I added just clocksource=tsc tsc=reliable but it didn't have any effect:

Code: 
root@pve01:~# dmesg|grep -E "clocksource|tsc|hpet"
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-7.0.14-3-pve root=/dev/mapper/pve-root ro quiet nomodeset clocksource=tsc tsc=reliable
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2654.631 MHz processor
[    0.015034] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.074706] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-7.0.14-3-pve root=/dev/mapper/pve-root ro quiet nomodeset clocksource=tsc tsc=reliable
[    0.283720] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417870 ns
[    0.288741] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2643d45021b, max_idle_ns: 440795315087 ns
[    0.472958] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.560918] tsc: Marking TSC unstable due to TSC halts in idle
[    0.560918] hpet: 4 channels of 0 reserved for per-cpu timers
[    0.560918] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31, 31
[    0.560918] hpet0: 4 comparators, 64-bit 25.000000 MHz counter
[    0.573348] clocksource: Switched to clocksource hpet
[    0.584347] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
root@pve01:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
hpet acpi_pm

My speculation is that the kernel's determination that it can reach power states in which TSC doesn't run and overrides whatever you've otherwise specified.

I did find some suggestions indicating that one could tell the kernel not to go into the problematic power states, so I now have:

Code: 
root@pve01:~# dmesg|grep -E "clocksource|tsc|hpet"
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-7.0.14-3-pve root=/dev/mapper/pve-root ro nomodeset clocksource=tsc tsc=reliable intel_idle.max_cstate=1 processor.max_cstate=1
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2654.636 MHz processor
[    0.015111] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.074799] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-7.0.14-3-pve root=/dev/mapper/pve-root ro nomodeset clocksource=tsc tsc=reliable intel_idle.max_cstate=1 processor.max_cstate=1
[    0.284066] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417870 ns
[    0.289092] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2643d89a49f, max_idle_ns: 440795285435 ns
[    0.476371] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.566268] hpet: 4 channels of 0 reserved for per-cpu timers
[    0.566268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31, 31
[    0.566268] hpet0: 4 comparators, 64-bit 25.000000 MHz counter
[    0.578595] clocksource: Switched to clocksource tsc-early
[    0.588781] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    1.660381] tsc: Refined TSC clocksource calibration: 2654.713 MHz
[    1.660435] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x26442187891, max_idle_ns: 440795251548 ns
[    1.660496] clocksource: Switched to clocksource tsc
root@pve01:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm
root@pve01:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

So I am running TSC on the new kernel now, at least temporarily. I have fired up the VMs and don't seem to have either CPU or I/O issues and it seems otherwise stable. If it seems OK in a few hours, I will add clocksource=tsc tsc=reliable intel_idle.max_cstate=1 processor.max_cstate=1 to my /etc/default/grub permanently (although I am sure some of that is redundant). After that, it looks safe to move on to the other nodes in the cluster, as they were already using TSC.

Thanks again for your help.
 
Last edited:
  • Like
Reactions: Onslow and Impact
Just for completeness, I will note that I only needed to add intel_idle.max_cstate=1 processor.max_cstate=1 to my kernel options to get the v7 kernel to choose TSC over HPERF; the clocksource=tsc tsc=reliable did not harm but turned out not to be required.
 
  • Like
Reactions: Impact
You must log in or register to reply here.
Top Bottom
Back