Pinning vCPU causes 100% cpu pressure stall, using isolcpus

N

ns33

Member
Apr 4, 2024
49
3
13
I have VM which runs an app requiring low latency as close to bare metal as possible. Note it is not a game but something doing compute. Maybe putting such in a virtual environment isn't the best scenario but I'm going to try

Quick system arch, each cluster node is dual socket, 2x 18c/36t intel cpu

I config the vm as followed for the cpu:
Code: 
Affinity: 0-15
cores: 16
cpu: host
cpuunits: 2000
numa: 1
numa0: cpus=0-15,hostnodes=1,memory=24579,policy=bind

From my understanding, even though the affinity is set for the VM cpu conf, the host might still schedule stuff on those cores/threads. In searching, I found I could take it a step further by isolating those physical core within grub using the arg
Code: 
isolcpus
. Which I did, and I can confirm they're isolated by
Code: 
cat /sys/devices/system/cpu/isolated
but when I boot the VM, which is Windows, I see the CPU stall get pegged at 100% the moment the windows boot loading screen appears (spinning circle) which basically comes to a standstill.

I also ran
Code: 
taskset -pc $(cat /var/run/qemu-server/<VMID>.pid)
and that shows the vm process id is tied to cpu cores 0-15.

If I remove the grub config line, but keep everything the same, my performance is better but I'm still getting cpu stall spike around 0.4%. Sure a tiny amount but no other vm in the cluster is reporting stalls. As a check, I shutdown every other vm on the node and ran this one standalone, still the same results with or without the grub line.

Am I doing something wrong? Am I missing a step? CPU and memory usage on the vm during running at stable with cpu just barely breaching 50% and RAM around 60%
 
That's isolcpus doing exactly what it says. It disables the scheduler's load balancing on those cores, and PVE's affinity: option only sets a mask on the whole qemu process. So all 16 vcpu threads end up on the same one or two cores and nothing ever moves them. Windows fires up all its vcpus at boot and they stall waiting on each other.

If you want isolcpus you have to pin each vcpu thread yourself:
- Start the VM
- Run qm monitor <vmid> then info cpus to get the thread ids, and taskset -pc <core> <tid> for each one.

A couple other things:
  • Check lscpu -e. Your memory is bound to hostnodes=1 but cpus 0-15 are probably node 0. Isolate the HT siblings of whatever cores you pick too, or the host will still run on the other half of each core.
  • Keep cpu 0 out of it. The kernel drops it from nohz_full anyway and a lot of default irqs land there.
  • Set affinity: to the cores you did NOT isolate. Only the pinned vcpu threads belong on the isolated ones. Best case, leave a couple of cores on the same numa node un-isolated and use those, so qemu's other threads stay close to the VM's memory.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
You must log in or register to reply here.
Top Bottom
Back