[SOLVED] Proxmox 8.0 / Kernel 6.2.x 100%CPU issue with Windows Server 2019 VMs

[jens-maus]

upgraded some different Servers to PVE8 (standalone/ceph-cluster and so on...) and the 2019-VMs (RDS mostly) runs better/faster than with PVE7.
The CPU´s are all differently, few old Xeons to modern Xeons, AMD Milans etc.....

Please share the concrete VM settings of your 2019VMs as I can hardly believe you are not having that issue when running PVE8+kernel6.2. So there must be some concrete differences in the VM config settings when they are working nicely for you. So please share.

The only one where they are similar, the CPU-Governor is always SCHEDUTIL.

We are always using the performance CPU governor, thus have cpu frequency scaling disabled, thus this can't be an explaination IMHO.

 

[ITT]

There´s a lot of 2019-VMs, but here one Example:

agent: 1
bios: ovmf
boot: order=scsi0;net0
cores: 12
cpu: host
cpulimit: 10
efidisk0: VM_NVMe:vm-2001-disk-3,efitype=4m,pre-enrolled-keys=1,size=528K
machine: pc-q35-7.2
memory: 65536
name: REMOVED
net0: virtio=42:12:95:A9:81:83,bridge=vmbr2000,queues=8
numa: 0
onboot: 1
ostype: win10
scsi0: VM_NVMe:vm-2001-disk-1,cache=writeback,discard=on,iothread=1,size=1T,ssd=1
scsi1: VM_NVMe:vm-2001-disk-2,cache=writeback,discard=on,iothread=1,size=1T,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=4cbdced0-1eb3-4803-8684-72fe785144ce
sockets: 1
vga: virtio
vmgenid: 1c34d235-b8b7-465f-87ab-bae83b6ab2f4

 

[jens-maus]

There´s a lot of 2019-VMs, but here one Example:

cores: 12
cpu: host
cpulimit: 10
memory: 65536
sockets: 1

You are "just" using 64GB of RAM and only a single CPU socket with 12 cores. In our recent tests it looks like this also affects the probability of hitting the issue. With increasing memory size (>128GB) as well as with CPU sockets > 1 the issue seem to manifest itself more prominent. In addition, why do you specify a cpulimit here? Perhaps this also affects the probability of being hit by the issue.

 

[ITT]

You are "just" using 64GB of RAM and only a single CPU socket with 12 cores. In our recent tests it looks like this also affects the probability of hitting the issue. With increasing memory size (>128GB) as well as with CPU sockets > 1 the issue seem to manifest itself more prominent. In addition, why do you specify a cpulimit here? Perhaps this also affects the probability of being hit by the issue.

"Just 64GB RAM" because its just an example-VM that doesn´t need more.
Single-Socket because the Hardware is on an AMD Milan, this is a Single-Socket-Blade with 48 Cores.
CPU-Limit -> simply to limit the VM

 

[Whatever]

"Just 64GB RAM" because its just an example-VM that doesn´t need more.
Single-Socket because the Hardware is on an AMD Milan, this is a Single-Socket-Blade with 48 Cores.
CPU-Limit -> simply to limit the VM

We also noticed that problem appears more when >64Gb ram allocated (especially 128 and more) and more than 24 vCPUs

P.S as I mentioned in another thread, setting "mitigations=off" helped (as workaround only!)

 

[ITT]

We also noticed that problem appears more when >64Gb ram allocated (especially 128 and more) and more than 24 vCPUs

P.S as I mentioned in another thread, setting "mitigations=off" helped (as workaround only!)

This size applies only to our Database Servers...
Hmm, mitigations=off is set up everywhere here.

 

[Whatever]

This size applies only to our Database Servers...
Hmm, mitigations=off is set up everywhere here.

R u sure it is correctly applied?
Check with lscpu. Should be like this:

root@063-pve-04347:~# lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  56
  On-line CPU(s) list:   0-55
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel
  Model name:            Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
    BIOS Model name:     Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz  CPU @ 2.6GHz
    BIOS CPU family:     179
    CPU family:          6
    Model:               79
    Thread(s) per core:  2
    Core(s) per socket:  14
    Socket(s):           2
    Stepping:            1
    CPU(s) scaling MHz:  75%
    CPU max MHz:         3500.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            5200.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx
                         fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts re
                         p_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx sm
                         x est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadli
                         ne_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 in
                         vpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_a
                         djust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cq
                         m_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   896 KiB (28 instances)
  L1i:                   896 KiB (28 instances)
  L2:                    7 MiB (28 instances)
  L3:                    70 MiB (2 instances)
NUMA:                   
  NUMA node(s):          2
  NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54
  NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55
Vulnerabilities:         
  Itlb multihit:         KVM: Vulnerable
  L1tf:                  Mitigation; PTE Inversion; VMX vulnerable
  Mds:                   Vulnerable; SMT vulnerable
  Meltdown:              Vulnerable
  Mmio stale data:       Vulnerable
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
  Spectre v2:            Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
  Srbds:                 Not affected
  Tsx async abort:       Vulnerable

 

[ITT]

R u sure it is correctly applied?

Searched an Host with the same CPU and here it is:

root@pbs-dc1:~# lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  28
  On-line CPU(s) list:   0-27
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel(R) Corporation
  Model name:            Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
    BIOS Model name:     Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz  CPU @ 2.6GHz
    BIOS CPU family:     179
    CPU family:          6
    Model:               79
    Thread(s) per core:  2
    Core(s) per socket:  14
    Socket(s):           1
    Stepping:            1
    CPU(s) scaling MHz:  43%
    CPU max MHz:         3500.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            5194.05
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc a
                         rch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dc
                         a sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin
                         ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt x
                         saveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   448 KiB (14 instances)
  L1i:                   448 KiB (14 instances)
  L2:                    3.5 MiB (14 instances)
  L3:                    35 MiB (1 instance)
NUMA:                   
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-27
Vulnerabilities:         
  Itlb multihit:         KVM: Vulnerable
  L1tf:                  Mitigation; PTE Inversion; VMX vulnerable
  Mds:                   Vulnerable; SMT vulnerable
  Meltdown:              Vulnerable
  Mmio stale data:       Vulnerable
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
  Spectre v2:            Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
  Srbds:                 Not affected
  Tsx async abort:       Vulnerable

 

[jens-maus]

Searched an Host with the same CPU and here it is:

root@pbs-dc1:~# lscpu
Vulnerabilities:        
  Itlb multihit:         KVM: Vulnerable
  L1tf:                  Mitigation; PTE Inversion; VMX vulnerable
  Mds:                   Vulnerable; SMT vulnerable
  Meltdown:              Vulnerable
  Mmio stale data:       Vulnerable
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
  Spectre v2:            Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
  Srbds:                 Not affected
  Tsx async abort:       Vulnerable

Sure, but you said you are NOT affected by the issue mentioned here, right? Then this would make perfectly sense since you have mitigations=off enabled accordingly.

 

[ITT]

Sure, but you said you are NOT affected by the issue mentioned here, right? Then this would make perfectly sense since you have mitigations=off enabled accordingly.

I hope so...

 

[croRF]

an example of one of our vms. has never fully locked up but cpu will top out at 100% every 2 seconds.
our vms having issues are only set to 1 socket. tried 'x86-64-v2-AES' and 'kvm64'
kernel 6.2.16
Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz (x2 two physical cpus but we only set vms to 1 socket)

agent: 1
bios: ovmf
boot: order=scsi0;ide2;scsi3
cores: 32
cpu: x86-64-v2-AES
efidisk0: vms_nvme:vm-18226-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
hotplug: disk,network,usb,memory,cpu
machine: pc-i440fx-7.0
memory: 40960
meta: creation-qemu=7.0.0,ctime=1663608323
name:
net0: virtio=B2:96:D8:03:5E:DC,bridge=vmbr0
numa: 1
ostype: win10
scsi0: vms_nvme:vm-18226-disk-1,cache=writeback,size=100G,ssd=1
scsi1: vms_nvme:vm-18226-disk-2,cache=writeback,discard=on,size=500G
scsihw: virtio-scsi-pci
smbios1: uuid=519c62b0-29c3-4471-a675-109f7845a117
sockets: 1
tags: terminal
tpmstate0: vms_nvme:vm-18226-disk-4,size=4M,version=v2.0
vcpus: 32
vmgenid: 2866d2b8-1daf-4d1a-8c9c-2ab15bd60e10

switch to kernel 5.15 and no issues.

 

[rj45]

Can you try to disable "nested virtualization" feature
for intel:
options kvm_intel nested=0
for amd:
options kvm_amd nested=0

 

[jens-maus]

switch to kernel 5.15 and no issues.

Please try to set mitigations=off as an additional kernel command option (in /etc/default/grub) and then run update-grub and boot kernel 6.2 again and see if the vm works more smoothly with mitigations disabled because there are reports that this improves things.

 

[Whatever]

Please try to set mitigations=off as an additional kernel command option (in /etc/default/grub) and then run update-grub and boot kernel 6.2 again and see if the vm works more smoothly with mitigations disabled because there are reports that this improves things.

For the record:
I had to do the following:

1. nano /etc/kernel/cmdfile -> root=ZFS=rpool/ROOT/pve-1 boot=zfs mitigations=off
2. proxmox-boot-tool refresh
3. reboot
4. Check with lscpu

 

[jens-maus]

For the record:
I had to do the following:

1. nano /etc/kernel/cmdfile -> root=ZFS=rpool/ROOT/pve-1 boot=zfs mitigations=off
2. proxmox-boot-tool refresh
3. reboot
4. Check with lscpu

As said before, the sequence I used to add the mitigations=off kernel option was:

1. add mitigations=off to the GRUB_CMDLINE_LINUX_DEFAULT variable in /etc/default/grub
2. run update-grub
3. reboot
4. check lscpu

And AFAIK this is usually a common way to apply additional kernel options to the used kernels in proxmox as well (cf. https://pve.proxmox.com/wiki/Host_Bootloader)

 

[jens-maus]

Now with having applied the mitigations=off kernel command option to our affected Proxmox8/kernel6.2 hosts I can support the observations of @Whatever that turning off CPU bug mitigations seem to solve the 100%CPU issue here. After having applied that (https://forum.proxmox.com/threads/p...th-windows-server-2019-vms.130727/post-574611) kernel option and having rebooted kernel 6.2 again, the windows2019 VMs work much more smoothly and without any more 100%CPU spikes.

So for us applying the mitigations=off kernel option seem to have immediately solved our issues (thanks @Whatever !)

However, I would of course be curious to see if this is also true for others in here suffering from the same issue and if the Proxmox developers have any explaination/solution for that. At least they should now have a hint where to look at regarding the 100%CPU spikes and VM stalling that seem to occur since kernel > 5.15.

 

[jens-maus]

Sorry to come back with some bad news here so early after having stated that using mitigations=off seem to have solved our issues. However, this does not seem to be completely the case as with increasing number of users logging into the windows2019 rds nodes with kernel 6.2+mitigations=off we are seeing again some 100%CPU spikes and ICMP ping request time increases and short Proxmox console stalling which we did not see yesterday evening after we tested this ourselves - but with only very low amount of load against the VM. But now with increasing load the high CPU spikes and ICMP ping time increases as well as Proxmox console stalling are back again, however quite more rarely. But the spikes are clearly visible here:


Please note that in this view "ts19d" and "ts19e" are the two nodes we just rebooted with kernel 6.2+mitigations=off. All others are still running kernel 5.15 which seem to work much more smoothly in the end. Clearly visible are the permanent CPU spikes which appear from time to time, which in the end also end up in high ICMP ping times and temporary Proxmox console stalling.

So sorry, but I think our issues with CPU spikes and kernel 6.2 and not fully solved yet, however, using mitigations=off seem to have improved the situations slightly since without that we have definitely more severe/prominent issues right from the beginning.

 

[rj45]

cat /sys/module/kvm_*/parameters/nested ?
if Y switch off and test

 

[Whatever]

Sorry to come back with some bad news here so early after having stated that using mitigations=off seem to have solved our issues. However, this does not seem to be completely the case as with increasing number of users logging into the windows2019 rds nodes with kernel 6.2+mitigations=off we are seeing again some 100%CPU spikes and ICMP ping request time increases and short Proxmox console stalling which we did not see yesterday evening after we tested this ourselves - but with only very low amount of load against the VM. But now with increasing load the high CPU spikes and ICMP ping time increases as well as Proxmox console stalling are back again, however quite more rarely. But the spikes are clearly visible here:

In our environment we so exactly the same. CPU spikes and ICMP ping losses + novnc console freeze with increasing number of RDP users logging into W2019. However setting mitigations=off helped (>50 users work fine on each W2019 vm)

One more interesting thing. We could not reproduce an issue (even with enabled mitigations) if we set 4-8 vCPUs and 16Gb RAM or less

 

[jens-maus]

In our environment we so exactly the same. CPU spikes and ICMP ping losses + novnc console freeze with increasing number of RDP users logging into W2019. However setting mitigations=off helped (>50 users work fine on each W2019 vm)

One more interesting thing. We could not reproduce an issue (even with enabled mitigations) if we set 4-8 vCPUs and 16Gb RAM or less

Yes, this matches our experience as well. We also don't see issues with VMs of low memory or less vCPUs or even CPU sockets assigned. But as we usually use one proxmox host for a single w2019 RDS server vm with full CPU and RAM capacity (>200GB RAM) the issue is quite prominent. As said earlier, things improved with the mitigations=off approach you suggested (thx!) but today as users logged in we can clearly see the issue again. Not so prominent and with less probability, but especially running a constant ICMP ping against a vm shows that from time to time ping times raise up to 10-20 seconds and then the novnc proxmox console freezes for some time, the proxmox host itself then shows also 100% cpu spikes and then after some seconds everything returns to normal but spikes come back after some time again. As said, with mitigations disabled it works better now, but not as smoothly as with kernel 5.15 booted. So there must be something else. So can you please share a VM config of one of your w2019 VMs so that we can compare.