[SOLVED] How to specify numa cpus in vm.conf?

S

SirMaster

Renowned Member
May 20, 2015
23
2
68
I'm sure this is a simple answer but for some reason I can't figure out what the documentation is telling me.

I'm trying to specify cores 0,2,4,6 out of my 0-7 cores. Currently my numa0 line looks like this:

numa0: cpus=0,hostnodes=0,memory=4096,policy=bind

I think this is valid so far, but I can't figure out how to specify additional cores. I've tried a bunch of different different things but clearly I cannot make sense of the documentation: cpus=<id[-id];...>
 
sigxcpu said:
I assume you do
Code: 
cpus=0;2;4;6
Click to expand...

That's what I thought, but it's not working.

I have:
Code: 
boot: cdn
bootdisk: virtio0
cores: 8
cpu: cputype=host,hidden=0
hostpci0: 01:00.0,pcie=1,x-vga=on
hostpci1: 01:00.1,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 4096
name: nick-htpc
net0: virtio=30:F7:D7:01:3D:83,bridge=vmbr0
numa: 1
numa0: cpus=0;2;4;6,hostnodes=0,memory=4096,policy=bind
ostype: win8
smbios1: uuid=4ad86913-60ba-4904-a4ab-330d07a8a638
sockets: 1
usb0: host=3-9.1.3
virtio0: ssd:105/vm-105-disk-1.qcow2,size=100G

And when I start the VM I get:
Code: 
kvm: -numa node,nodeid=0,cpus=0,2,4,6,memdev=ram-node0: Invalid parameter '2'
 
spirit said:
you need to user ";" not "," to separate values
Click to expand...

I did. Did you see my attached conf file?

My line is:
Code: 
numa0: cpus=0;2;4;6,hostnodes=0,memory=4096,policy=bind

The commas show up only in the error for whatever reason.
 
Last edited:
Yeah, parsing the semi colons wrong seems like a real possibility.

Thanks and there is not necessarily a great rush on this.
 
Ok, I have found the bug. It's a proxmox bug.

in kvm, the cpus option need to be used multiple time when we have a list

"-numa node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,memdev=ram-node0"


I have sent this patch to mailing list.

Code: 
From 01bb3e836bd2419a15cc3b65e933fab77c4c5f0a Mon Sep 17 00:00:00 2001
From: Alexandre Derumier <aderumier@odiso.com>
Date: Mon, 10 Oct 2016 09:35:47 +0200
Subject: [PATCH] numaX : use cpus option multiple time if cpulist

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer/Memory.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/PVE/QemuServer/Memory.pm b/PVE/QemuServer/Memory.pm
index 087584b..0e4c830 100644
--- a/PVE/QemuServer/Memory.pm
+++ b/PVE/QemuServer/Memory.pm
@@ -256,7 +256,7 @@ sub config {
            # cpus
            my $cpulists = $numa->{cpus};
            die "missing NUMA node$i cpus\n" if !defined($cpulists);
-           my $cpus = join(',', map {
+           my $cpus = join(',cpus=', map {
                my ($start, $end) = @$_;
                defined($end) ? "$start-$end" : $start
            } @$cpulists);
--
 
So I see the update for this was released a few days ago and it seems to work.

However maybe I am misunderstanding how the numa configuration works, but my goal was assign cores 0,2,4,6 (the 4 real non-hyperthreaded cores) from my 0-7 cores to my VM which I wanted to assign 4 internal cores to that VM. So that the VM would only ever run on real cores, not hyperthreaded cores.

However, when I use this line now:
Code: 
numa0: cpus=0;2;4;6,hostnodes=0,memory=4096,policy=bind

This works, but only when I have my VM's core count set to 7 or 8. If I set my VM core count to 4 I get this error when starting it.
Code: 
kvm: -numa node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,memdev=ram-node0: CPU index (4) should be smaller than maxcpus (4)

Code: 
TASK ERROR: start failed: command '/usr/bin/kvm -id 105 -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/105.pid -daemonize -smbios 'type=1,uuid=4ad86913-60ba-4904-a4ab-330d07a8a638' -name nick-htpc -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off' -m 4096 -object 'memory-backend-ram,id=ram-node0,size=4096M,host-nodes=0,policy=bind' -numa 'node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,memdev=ram-node0' -k en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on' -device 'vfio-pci,host=01:00.1,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0' -device 'usb-host,hostbus=3,hostport=9.1.3,id=usb0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:e9d3cdc1e44' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/mnt/misc/proxmox/images/105/vm-105-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap105i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=30:F7:D7:01:3D:83,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1

Why does the maxcpus count need to be set higher than the index of the highest core number I'm specifying in numa?

If I change my VM config to 1 core and my numa line to just use core 7 for instance like this:
Code: 
numa0: cpus=7,hostnodes=0,memory=4096,policy=bind

Then I will get the same error.
Code: 
kvm: -numa node,nodeid=0,cpus=7,memdev=ram-node0: CPU index (7) should be smaller than maxcpus (1)

I would have to give my VM 8 cores in order to boot it even though I just want to specify 1 specific core.

I understand there is generic CPU pinning from the host, but I wanted my VMs to always be pinned to specific cores without having to re-pin then manually every time they are restarted. It seemed like numa does this, but maybe I am going about this completely wrong.
 
Last edited:
  • Like
Reactions: Thomas M. Berger
What are the exact VM settings to get the VM to even boot?
I've literally copy pasted this into the vm.conf and it won't take.
Code: 
numa0: cpus=0;2;4;6,hostnodes=0,memory=4096,policy=bind

8 cores
4 vcpu
numa enabled


I've noticed toggling hotplug cpu and memory doesn't play well so thats off.

Code: 
pvedaemon[21495]: start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=691058c4-d0d1-4039-8b28-a256a9cb5613' -name dmoraTest -smp '4,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -object 'memory-backend-ram,id=ram-node0,size=4096M,host-nodes=0,policy=bind' -numa 'node,nodeid=0,cpus=0,1,2,3,memdev=ram-node0' -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:8638c9979685' -drive 'file=/localdata/template/iso/CentOS-7-x86_64-Minimal-1511.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=rbd:vmStoragePool/vm-100-disk-1:mon_host=172.16.0.1;172.16.0.2;172.16.0.3:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/ceph_storage.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B6:DF:92:7A:D7:8B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1

root@prx026:~# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 15933 MB
node 0 free: 4868 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 16125 MB
node 1 free: 764 MB
node distances:
node  0  1
  0:  10  21
  1:  21  10

bootdisk: scsi0
cores: 8
hotplug: disk,network,usb,cpu
ide2: localdata:iso/CentOS-7-x86_64-Minimal-1511.iso,media=cdrom
memory: 4096
name: dmoraTest
net0: virtio=B6:DF:92:7A:D7:8B,bridge=vmbr0,tag=210
numa: 1
numa0: cpus=0;1;2;3,hostnodes=0,memory=4096,policy=bind
ostype: l26
scsi0: ceph_storage:vm-100-disk-1,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=691058c4-d0d1-4039-8b28-a256a9cb5613
sockets: 1
vcpus: 4
 
You must log in or register to reply here.
Top Bottom