VM soft-locking up with ceph during disk benchmark

[anaxagoras]

I have a windows VM and i'm running crystal disk benchmark in it, and during the benchmark the VM does a soft-lockup. This is repeatable in my environment. I'm running the latest version of PVE 8.1.11, kernel 6.5.13-5-pve, and ceph reef version 18.2.2


What i mean by a soft lockup is: If task manager is open you can see it's running, the mouse cursor still moves, and i can see some sections on the screen react to a mouseover, but the system is not responsive in any other way, can't click on anything, or interact with the keyboard. It almost feels to me like the VM is losing access to the disk for an instant and windows doesn't know what to do.

I've done ceph rados benchmarks with no issues.

I'm unsure if this is a ceph issue, proxmox issue, windows issue, crystal disk benchmark issue, or other issue with my environment. Being new to ceph, and relatively new to proxmox i'm unsure what logs i should be looking at to try and identify what's going on.

 

[mgabriel]

Please post your VM config. You may try to set your IO scheduler from io_uring to native for this VM and test again. May help, may not ;-). Please report back if possible.

 

[gfngfn256]

If you want to eliminate Ceph as the culprit, if possible migrate/copy/create same VM on non-Ceph storage & test again.

 

[anaxagoras]

If you want to eliminate Ceph as the culprit, if possible migrate/copy/create same VM on non-Ceph storage & test again.

So i did do this test under zfs before setting up ceph, i was curious to see the performance difference to a VM. It worked fine.

Please post your VM config. You may try to set your IO scheduler from io_uring to native for this VM and test again. May help, may not ;-). Please report back if possible.

Spoiler: VM Config

root@pve4:/etc/pve/nodes/pve4/qemu-server# cat 111.conf 
agent: 1,fstrim_cloned_disks=1
audio0: device=ich9-intel-hda,driver=spice
balloon: 4096
bios: ovmf
boot: order=scsi0;ide0;ide2;net0
cores: 4
cpu: host
efidisk0: vm-disks:vm-111-disk-1,efitype=4m,pre-enrolled-keys=1,size=528K
ide0: none,media=cdrom
ide2: none,media=cdrom
machine: pc-q35-8.1
memory: 16384
meta: creation-qemu=8.1.5,ctime=1710098953
name: Windows11Test
net0: virtio=BC:24:11:EA:F3:23,bridge=vmbr1,firewall=1,queues=4
numa: 0
ostype: win11
scsi0: vm-disks:vm-111-disk-0,cache=writeback,discard=on,iothread=1,size=52G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=2824745d-2a1e-4124-a9db-a6a8fe1ac8d1
sockets: 1
spice_enhancements: foldersharing=1,videostreaming=all
tablet: 0
tpmstate0: vm-disks:vm-111-disk-2,size=4M,version=v2.0
vga: qxl
vmgenid: c00e4056-f1ed-4ae3-af27-b5efb38ce0f3

 

[anaxagoras]

Please post your VM config. You may try to set your IO scheduler from io_uring to native for this VM and test again. May help, may not ;-). Please report back if possible.

I changed the io scheduler to native and it died again. it seems to keep dying on the random read tests. it seems to be getting through the sequential 1M Q8Tq, and the SEQ128kQ32T1. It keeps dying on either Random 4k Q32T16 or RND4kQ1T1. i'm unsure if that has any bearing.

It doesn't make it to any of the write tests

 

[anaxagoras]

I made some progress on this.

If i do a less aggressive benchmark using the "default" profile don't get a lockup. SEQ1M Q8T1, SEQ1M Q1T1, RND4K 32T1, RND4K Q1T1 it runs fine.

The more aggressive "ssd" profile is where it has a problem. SEQ1M Q8TQ, SEQ128K Q32T1, RND4K Q32T16, RND4KQ1T1.

so it seems to die pretty consistently on the Random Read Q32 T16 test. I'm imagining this is just somehow hitting some kind of thread limit that's causing something to be unhappy. Now this is a synthetic test and i'll unlikely ever hit that kind of workload, but I'm concerned by way it's failing.

 

[anaxagoras]

hmmm, i'm seeing errors in dmesg around the time the system locks up. i'm sure there's a relation.

[65039.247828] x86/split lock detection: #AC: CPU 0/KVM/623152 took a split_lock trap at address: 0xfffff8052744bb6d

 

[gfngfn256]

x86/split lock detection

Read this which gives a very basic understanding of the problem. Based on that, its likely this is what you are suffering under a heavy crystal disk benchmark.

I've never suffered it, so you may want to do your own research.

 

[VictorSTS]

There's also a new page in the wiki with all info about split lock.
Never had this on any system, so couldn't test many thins. You may try to enable NUMA in CPU settings of the VM and check if there's any difference?

 

[gfngfn256]

There's also a new page in the wiki with all info about split lock.

Thanks for that. Its actually quite thorough & includes mitigation/s.

 

[garfield659]

Were you able to get it resolved and if so how? I am running into the exact same issue, windows 2022 server running as a VM on Proxmox with CEPH as the underlining datastore.