TASK ERROR: total memory for NUMA nodes must be equal to vm static memory

[hanzfritz85]

VM conf

bootdisk: scsi0
cores: 4
cpu: host
hotplug: disk,network,usb,memory,cpu
ide2: none,media=cdrom
memory: 4096
name: debian
numa: 1
numa0: cpus=0-3,hostnodes=0,policy=bind,memory=4096
onboot: 1
ostype: l26
scsi0: local:101/vm-101-disk-0.raw,size=25G
scsihw: virtio-scsi-pci
serial0: socket
sockets: 1
vcpus: 4

it simply won't start with the above error.
What am I doing wrong?

 

[Alwin]

numa0: cpus=0-3,hostnodes=0,policy=bind,memory=4096

Does the NUMA node provide 4GB of memory?

 

[hanzfritz85]

Started with 1G memory

available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64288 MB
node 0 free: 56078 MB
node distances:
node 0
0: 10

 

[Alwin]

available: 1 nodes (0)

Is it a NUMA system at all?

KVM tries to schedule its threads onto the cores of the same NUMA node. This is already achieved by setting NUMA on the VMs CPU.

 

[hanzfritz85]

oh
it's a single socket server
is it possible to pin cpu cores for vm on a non-numa system?

 

[hanzfritz85]

and why did it start with 1GB memory and сpu pin enabled

 

[Stefan_R]

I just checked our code and it might be a bug on our side, I'm investigating. For the time being, try your original config but disable memory hotplug, that works on my end. Do you have a use-case for a custom NUMA topology AND memory hotplugging enabled?

 

[hanzfritz85]

I just checked our code and it might be a bug on our side, I'm investigating. For the time being, try your original config but disable memory hotplug, that works on my end. Do you have a use-case for a custom NUMA topology AND memory hotplugging enabled?

yep. it starts with memory hotplugging disabled and any amount of memory
with cpu core pin config
is it a bug?

 

[Stefan_R]

yep. it starts with memory hotplugging disabled and any amount of memory
with cpu core pin config
is it a bug?

Kind of. It turns out memory hotplug with a custom numa topology doesn't work in general with how PVE handles it. I've sent a patch to our mailing list to disable it with a more useful error message.

Is there a use-case where you would actually need this (hotplug + custom NUMA topo)?

 

[hanzfritz85]

Kind of. It turns out memory hotplug with a custom numa topology doesn't work in general with how PVE handles it. I've sent a patch to our mailing list to disable it with a more useful error message.

thank you!

Is there a use-case where you would actually need this (hotplug + custom NUMA topo)?

yes, the idea is to have a specific VM with dedicated cores of maximum possible performance per core. so nothing expected to share it. other vms will be assigned to other cores. and memory hotplug is a default, but i suppose is less important in my case.

 

[hanzfritz85]

I don't see any effect of pinned cores. Core load being monitored with htop on proxmox migrates between cores.
Can anyone clarify to me what is the correct behavior? Maybe in other hypervisors?

 

[Stefan_R]

Using the numa options is not really "CPU pinning" in the traditional sense, it just means that the cores specified will be migrated only to physical CPUs in on specific NUMA node. But within that NUMA node the processes can be migrated between cores freely.

See also this recent thread about actual CPU pinning.

 

[hanzfritz85]

Using the numa options is not really "CPU pinning" in the traditional sense, it just means that the cores specified will be migrated only to physical CPUs in on specific NUMA node. But within that NUMA node the processes can be migrated between cores freely.

See also this recent thread about actual CPU pinning.

Thank you for clarification. That's exactly what I wanted to achive

 

[pieteras.meyer]

Hi Stefan

Im also getting this error

TASK ERROR: total memory for NUMA nodes must be equal to vm static memory

proxmox-ve: 6.4-1 (running kernel: 5.11.7-1-pve)
pve-manager: 6.4-5 (running version: 6.4-5/6c7bf5de)
pve-kernel-5.11: 7.0-0+3~bpo10
pve-kernel-5.4: 6.4-1
pve-kernel-helper: 6.4-1
pve-kernel-5.11.7-1-pve: 5.11.7-1~bpo10
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-2
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-1
libpve-network-perl: 0.5-2
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.3-1
proxmox-backup-client: 1.1.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-3
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

 

[Stefan_R]

For the future, it is usually a good idea to open a new thread, instead of pinging one that's more than a year old

The error you posted can have many different causes. Can you please also post a description of what you're trying to do, your VM config ('qm config <vmid>') and the output of 'numactl -s' and 'free'?

 

[pieteras.meyer]

For the future, it is usually a good idea to open a new thread, instead of pinging one that's more than a year old

The error you posted can have many different causes. Can you please also post a description of what you're trying to do, your VM config ('qm config <vmid>') and the output of 'numactl -s' and 'free'?

Hi Stefan,

I have managed to resolve my issue, but for Interest stake below is my details,

We had to virtualize a router due to a lack of Driver support for Intel X710 cards in the software's kernel otherwise we would run it on baremetal.
the Host only runs 1 VM. After many hours of testing below is the best performing config I got working.

I still want to figure out how I can prevent the numa from interleave hits if anyone has ideas I think this could improve performance further

on this particular node we are using a AMD ROME 7302P CPU with 128GB ram.
the 7302P has 8 numa nodes with 2 cores in each node.

numactl -s
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
cpubind: 0 1 2 3 4 5 6 7
nodebind: 0 1 2 3 4 5 6 7
membind: 0 1 2 3 4 5 6 7

numactl -H
available: 8 nodes (0-7)
node 0 cpus: 0 1
node 0 size: 15905 MB
node 0 free: 15063 MB
node 1 cpus: 2 3
node 1 size: 16120 MB
node 1 free: 14995 MB
node 2 cpus: 4 5
node 2 size: 16120 MB
node 2 free: 15122 MB
node 3 cpus: 6 7
node 3 size: 16120 MB
node 3 free: 15173 MB
node 4 cpus: 8 9
node 4 size: 16120 MB
node 4 free: 15269 MB
node 5 cpus: 10 11
node 5 size: 16120 MB
node 5 free: 15162 MB
node 6 cpus: 12 13
node 6 size: 16120 MB
node 6 free: 15242 MB
node 7 cpus: 14 15
node 7 size: 16097 MB
node 7 free: 13676 MB
node distances:
node   0   1   2   3   4   5   6   7
  0:  10  11  11  11  11  11  11  11
  1:  11  10  11  11  11  11  11  11
  2:  11  11  10  11  11  11  11  11
  3:  11  11  11  10  11  11  11  11
  4:  11  11  11  11  10  11  11  11
  5:  11  11  11  11  11  10  11  11
  6:  11  11  11  11  11  11  10  11
  7:  11  11  11  11  11  11  11  10
 
 
  numastat
                           node0           node1           node2           node3
numa_hit                14708628        17688608        15102832        22616029
numa_miss                      0               0               0               0
numa_foreign                   0               0               0               0
interleave_hit             11848           11857           11847           11850
local_node              14701663        17666567        15082962        22593939
other_node                  6965           22041           19870           22090

                           node4           node5           node6           node7
numa_hit                12820127        19197264         9220122        20662500
numa_miss                      0               0               0               0
numa_foreign                   0               0               0               0
interleave_hit             11846           11830           11850           11842
local_node              12798372        19176543         9198363        20640929
other_node                 21755           20721           21759           21571

VM Config below

qm config 100
agent: 1
balloon: 0
boot: order=virtio0
cores: 16
cpu: host
hotplug: disk,network
machine: q35
memory: 8192
name: core01-chr
net0: virtio=52:54:00:B0:55:86,bridge=vmbr0,queues=16
net1: virtio=52:54:00:B1:9F:1D,bridge=vmbr10,queues=16
net2: virtio=52:54:00:84:D2:C3,bridge=vmbr11,queues=16
numa: 1
numa0: cpus=0-1,hostnodes=0,memory=1024,policy=bind
numa1: cpus=2-3,hostnodes=1,memory=1024,policy=bind
numa2: cpus=4-5,hostnodes=2,memory=1024,policy=bind
numa3: cpus=6-7,hostnodes=3,memory=1024,policy=bind
numa4: cpus=8-9,hostnodes=4,memory=1024,policy=bind
numa5: cpus=10-11,hostnodes=5,memory=1024,policy=bind
numa6: cpus=12-13,hostnodes=6,memory=1024,policy=bind
numa7: cpus=14-15,hostnodes=7,memory=1024,policy=bind
onboot: 0
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=afe21a5c-c9fc-49c1-83de-60d91a23f98b
sockets: 1
tablet: 0
virtio0: local-lvm:vm-100-disk-0,size=1152M
vmgenid: 039a640f-d43e-46d2-9757-b47b8105d695