Correct VM NUMA config on 2 sockets HOST

[Whatever]

Good day

Help me figure out and implement the correct virtual machine configuration for a dual-socket motherboard (PVE 8.4, 6.8.12 kernel)

Given:

root@pve-node-04840:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 104
On-line CPU(s) list: 0-103
Vendor ID: GenuineIntel
BIOS Vendor ID: Intel(R) Corporation
Model name: Montage Jintide(R) C6230R
...
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 1.6 MiB (52 instances)
L1i: 1.6 MiB (52 instances)
L2: 52 MiB (52 instances)
L3: 71.5 MiB (2 instances)
NUMA:
NUMA node(s): 2
NUMA node0 CPU(s): 0-25,52-77
NUMA node1 CPU(s): 26-51,78-103

NUMA topology

root@pve-node-04840:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77
node 0 size: 192047 MB
node 0 free: 58672 MB
node 1 cpus: 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103
node 1 size: 193512 MB
node 1 free: 90358 MB
node distances:
node 0 1
0: 10 21
1: 21 10

According to the PVE docs, I configured the VM as follows:

cores: 8
sockets: 2
memory: 65536
numa: 1
numa0: cpus=0-3;52-55,hostnodes=0,memory=32768,policy=bind
numa1: cpus=26-29;78-81,hostnodes=1,memory=32768,policy=bind

but got an error:

root@pve-node-04840:~# qm start 2802
kvm: -numa node,nodeid=0,cpus=0-3,cpus=52-55,memdev=ram-node0: CPU index (52) should be smaller than maxcpus (16)
start failed: QEMU exited with code 1

Based on the error text, I concluded that numaN describes the NUMA nodes for the guest system and is not directly linked to the NUMA nodes and CPU cores on the host.

In the PVE documentation, I found the 'affinity' option and changed the VM config to:

cores: 8
sockets: 2
memory: 65536
numa: 1
numa0: cpus=0-7,hostnodes=0,memory=32768,policy=bind
numa1: cpus=8-15,hostnodes=1,memory=32768,policy=bind
affinity: 0-3,52-55,26-29,78-81

The virtual machine started. But how do I link the affinity and the NUMA nodes of the virtual machine, and is it even possible?

From my point of view, maximum performance will be achieved when:

numa0: cpus=0-7,hostnodes=0   --> affinity: 0-3,52-55 (0-3 physycal cores, 52-55 HT cores, host NUMA node 0)
numa1: cpus=8-15,hostnodes=1   --> affinity: 26-29,78-81 (26-29 physycal cores, 78-81 HT cores, host NUMA node 1)

But I can't figure out how to achieve such a distribution.

Any help appreciated
Thank you in advance!

 

[readyspace]

Hi, in numa0/numa1, the cpus= list refers to guest vCPU indexes (0…vCPUs-1), not host CPU IDs.

Affinity is the host cpuset for the whole VM process (all vCPU threads), not per-vCPU.

Proxmox VE (QEMU) doesn’t expose per-vCPU pinning in the VM config, so you can’t directly “tie” guest vNUMA 0 to a subset of host CPUs and guest vNUMA 1 to another subset inside the same VM. You can only give QEMU a host CPU allowlist (via affinity) and let the scheduler/NUMA balancing keep threads local to their memory nodes.

You created a 16-vCPU VM (2 sockets × 8 cores × 1 thread = 16). In numa0 you set cpus=0-3;52-55.
But cpus= there must be guest vCPU indexes in the range 0…15. Index 52 is out of range, hence the error.

 

[Whatever]

Thank

Hi, in numa0/numa1, the cpus= list refers to guest vCPU indexes (0…vCPUs-1), not host CPU IDs.

Affinity is the host cpuset for the whole VM process (all vCPU threads), not per-vCPU.

Proxmox VE (QEMU) doesn’t expose per-vCPU pinning in the VM config, so you can’t directly “tie” guest vNUMA 0 to a subset of host CPUs and guest vNUMA 1 to another subset inside the same VM. You can only give QEMU a host CPU allowlist (via affinity) and let the scheduler/NUMA balancing keep threads local to their memory nodes.

It's all clear now. Thanks.