Kvm issue cpu Xeon Gold

[Oasisnet77]

Hi all,
i am facing a strange behaviour in a fresh installation Proxmox istance version 8.2.2
the host server use a 64 x Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (2 Sockets) and 128 GB ram
6 ssd in zfs raidz-2
i've imported a windows server 2019 virtual machine using 28 core per socket (use cpu host) and 64 GB ram and it's running extremely low.
launching atop command on proxmox shell i can see that the main process using resources is kvm and the host cpu indicates an average use > 200%
the same vm on an older server works normally

i think thas is a kernel problem related to that cpu, so i tried to use no-subscription repository with a new kernel but nothing changes.

now i would like to try an older kernel version to test the performance

are there any known issues related Xeon Gold 6326 CPU?

thanks
kind regards
Emanuele

 

[jsterr]

Hi Emanuele,

can you post the vmconfig, can be done via cli with root@PMX8:~# qm config 300 .

Thanks!

 

[VictorSTS]

Check the CPU speed in the host:

cat /proc/cpuinfo | grep MHz

Some BIOS settings may force the CPU to run at very low speed, and processes need much longer real time to execute. I would check the power/performance settings in BIOS and set it to something similar to "performance" or "balanced performance". I've also seen this behavior on multi PSU chassis when one PSU is missing/disconnected/faulty.

Also, defining "slow" in your case (i.e. what behavior you see) could give some pointers to the cause.

 

[Oasisnet77]

Hi,
thanks for your reply

here the output:
root@srv1:~# qm config 201
balloon: 0
bios: ovmf
boot: order=virtio0
cores: 24
cpu: host
efidisk0: VM_Pool:vm-201-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
machine: pc-i440fx-8.1
memory: 65536
meta: creation-qemu=8.1.5,ctime=1719401172
name: SistemaFProduzione
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=b8347957-6000-4496-8418-14eb3c29ea91
sockets: 2
tpmstate0: VM_Pool:vm-201-disk-1,size=4M,version=v2.0
vga: virtio,memory=512
virtio0: VM_Pool:vm-201-disk-2,cache=directsync,iothread=1,size=301G
virtio1: VM_Pool:vm-201-disk-3,cache=directsync,iothread=1,size=300G
vmgenid: 282e643b-28d4-461d-b870-e383d6695cda


meanwhile i tried lots of test and what i found extremely interesting is this log found in the syslog:

proxmox kernel: split_lock_warn: 32 callbacks suppressed

searching online i found this recent post updated by Proxmox, but can't find any solution
https://pve.proxmox.com/wiki/Split_lock_detection

seems something regarding kernel and cpu.

thanks
kind regards
Emanuele

 

[Oasisnet77]

Hi Victor
here the output requested

root@srv1:~# cat /proc/cpuinfo | grep MHz
cpu MHz : 800.160
cpu MHz : 800.195
cpu MHz : 800.188
cpu MHz : 800.061
cpu MHz : 2900.000
cpu MHz : 2900.000
cpu MHz : 800.280
cpu MHz : 2900.000
cpu MHz : 800.000
cpu MHz : 2900.000
cpu MHz : 800.162
cpu MHz : 800.108
cpu MHz : 800.106
cpu MHz : 2900.000
cpu MHz : 800.195
cpu MHz : 2900.000
cpu MHz : 800.104
cpu MHz : 799.945
cpu MHz : 800.120
cpu MHz : 800.160
cpu MHz : 800.234
cpu MHz : 800.271
cpu MHz : 800.186
cpu MHz : 800.309
cpu MHz : 800.000
cpu MHz : 800.000
cpu MHz : 800.204
cpu MHz : 800.124
cpu MHz : 800.337
cpu MHz : 800.167
cpu MHz : 800.094
cpu MHz : 800.112
cpu MHz : 800.140
cpu MHz : 2900.000
cpu MHz : 2900.000
cpu MHz : 2900.000
cpu MHz : 800.066
cpu MHz : 800.034
cpu MHz : 800.054
cpu MHz : 800.008
cpu MHz : 800.807
cpu MHz : 800.128
cpu MHz : 800.000
cpu MHz : 2900.000
cpu MHz : 2900.000
cpu MHz : 800.140
cpu MHz : 2900.000
cpu MHz : 800.121
cpu MHz : 800.135
cpu MHz : 800.079
cpu MHz : 800.113
cpu MHz : 800.000
cpu MHz : 800.045
cpu MHz : 800.044
cpu MHz : 800.091
cpu MHz : 800.180
cpu MHz : 800.070
cpu MHz : 800.135
cpu MHz : 799.955
cpu MHz : 800.111
cpu MHz : 800.046
cpu MHz : 800.275
cpu MHz : 800.091
cpu MHz : 800.217

bios is setted in efficiency favour performance.
i tried to switch on performance and it works !!!

thanks for your tip, it was the first time facing this issue but it was the first time using a Lenovo server

tahnks to all

 

[VictorSTS]

proxmox kernel: split_lock_warn: 32 callbacks suppressed

That's a problem with some app running on that VM (Firebird DB by chance?) that will make the kernel punish such misbehaving apps by slowing them down artificially. That very page has the possible options [1] to sort out the problem but be aware of the implications.

Such apps will reduce the performance of the whole server even if you disable split_lock_detection due to the fact that memory accesess has to be blocked completely for the app to access two cache lines at once.

i tried to switch on performance and it works !!!

Glad to know, but this just helped to make the split_lock issue less prevalent by making the CPU running faster more often. Might be good enough for your workload, though.

[1] https://pve.proxmox.com/wiki/Split_lock_detection#Options

 

[Oasisnet77]

I can confifirm that the split_lock issue is still present but now vm runs really more faster.
before your suggestion i tried to disable split_lock_detection but nothing changed.

for sure it depends by the customer application so for now your tip is enough to solve the problem.

thanks again
Emanuele