[SOLVED] How to specify numa cpus in vm.conf?

S

SirMaster

Renowned Member
May 20, 2015
23
2
68
I'm sure this is a simple answer but for some reason I can't figure out what the documentation is telling me.

I'm trying to specify cores 0,2,4,6 out of my 0-7 cores. Currently my numa0 line looks like this:

numa0: cpus=0,hostnodes=0,memory=4096,policy=bind

I think this is valid so far, but I can't figure out how to specify additional cores. I've tried a bunch of different different things but clearly I cannot make sense of the documentation: cpus=<id[-id];...>
 
sigxcpu said:
I assume you do
Code: 
cpus=0;2;4;6
Click to expand...

That's what I thought, but it's not working.

I have:
Code: 
boot: cdn
bootdisk: virtio0
cores: 8
cpu: cputype=host,hidden=0
hostpci0: 01:00.0,pcie=1,x-vga=on
hostpci1: 01:00.1,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 4096
name: nick-htpc
net0: virtio=30:F7:D7:01:3D:83,bridge=vmbr0
numa: 1
numa0: cpus=0;2;4;6,hostnodes=0,memory=4096,policy=bind
ostype: win8
smbios1: uuid=4ad86913-60ba-4904-a4ab-330d07a8a638
sockets: 1
usb0: host=3-9.1.3
virtio0: ssd:105/vm-105-disk-1.qcow2,size=100G

And when I start the VM I get:
Code: 
kvm: -numa node,nodeid=0,cpus=0,2,4,6,memdev=ram-node0: Invalid parameter '2'
 
spirit said:
you need to user ";" not "," to separate values
Click to expand...

I did. Did you see my attached conf file?

My line is:
Code: 
numa0: cpus=0;2;4;6,hostnodes=0,memory=4096,policy=bind

The commas show up only in the error for whatever reason.
 
Last edited:
Yeah, parsing the semi colons wrong seems like a real possibility.

Thanks and there is not necessarily a great rush on this.
 
Ok, I have found the bug. It's a proxmox bug.

in kvm, the cpus option need to be used multiple time when we have a list

"-numa node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,memdev=ram-node0"


I have sent this patch to mailing list.

Code: 
From 01bb3e836bd2419a15cc3b65e933fab77c4c5f0a Mon Sep 17 00:00:00 2001
From: Alexandre Derumier <aderumier@odiso.com>
Date: Mon, 10 Oct 2016 09:35:47 +0200
Subject: [PATCH] numaX : use cpus option multiple time if cpulist

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer/Memory.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/PVE/QemuServer/Memory.pm b/PVE/QemuServer/Memory.pm
index 087584b..0e4c830 100644
--- a/PVE/QemuServer/Memory.pm
+++ b/PVE/QemuServer/Memory.pm
@@ -256,7 +256,7 @@ sub config {
            # cpus
            my $cpulists = $numa->{cpus};
            die "missing NUMA node$i cpus\n" if !defined($cpulists);
-           my $cpus = join(',', map {
+           my $cpus = join(',cpus=', map {
                my ($start, $end) = @$_;
                defined($end) ? "$start-$end" : $start
            } @$cpulists);
--
 
So I see the update for this was released a few days ago and it seems to work.

However maybe I am misunderstanding how the numa configuration works, but my goal was assign cores 0,2,4,6 (the 4 real non-hyperthreaded cores) from my 0-7 cores to my VM which I wanted to assign 4 internal cores to that VM. So that the VM would only ever run on real cores, not hyperthreaded cores.

However, when I use this line now:
Code: 
numa0: cpus=0;2;4;6,hostnodes=0,memory=4096,policy=bind

This works, but only when I have my VM's core count set to 7 or 8. If I set my VM core count to 4 I get this error when starting it.
Code: 
kvm: -numa node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,memdev=ram-node0: CPU index (4) should be smaller than maxcpus (4)

Code: 
TASK ERROR: start failed: command '/usr/bin/kvm -id 105 -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/105.pid -daemonize -smbios 'type=1,uuid=4ad86913-60ba-4904-a4ab-330d07a8a638' -name nick-htpc -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off' -m 4096 -object 'memory-backend-ram,id=ram-node0,size=4096M,host-nodes=0,policy=bind' -numa 'node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,memdev=ram-node0' -k en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on' -device 'vfio-pci,host=01:00.1,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0' -device 'usb-host,hostbus=3,hostport=9.1.3,id=usb0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:e9d3cdc1e44' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/mnt/misc/proxmox/images/105/vm-105-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap105i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=30:F7:D7:01:3D:83,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1

Why does the maxcpus count need to be set higher than the index of the highest core number I'm specifying in numa?

If I change my VM config to 1 core and my numa line to just use core 7 for instance like this:
Code: 
numa0: cpus=7,hostnodes=0,memory=4096,policy=bind

Then I will get the same error.
Code: 
kvm: -numa node,nodeid=0,cpus=7,memdev=ram-node0: CPU index (7) should be smaller than maxcpus (1)

I would have to give my VM 8 cores in order to boot it even though I just want to specify 1 specific core.

I understand there is generic CPU pinning from the host, but I wanted my VMs to always be pinned to specific cores without having to re-pin then manually every time they are restarted. It seemed like numa does this, but maybe I am going about this completely wrong.
 
Last edited:
  • Like
Reactions: Thomas M. Berger
What are the exact VM settings to get the VM to even boot?
I've literally copy pasted this into the vm.conf and it won't take.
Code: 
numa0: cpus=0;2;4;6,hostnodes=0,memory=4096,policy=bind

8 cores
4 vcpu
numa enabled


I've noticed toggling hotplug cpu and memory doesn't play well so thats off.

Code: 
pvedaemon[21495]: start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=691058c4-d0d1-4039-8b28-a256a9cb5613' -name dmoraTest -smp '4,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -object 'memory-backend-ram,id=ram-node0,size=4096M,host-nodes=0,policy=bind' -numa 'node,nodeid=0,cpus=0,1,2,3,memdev=ram-node0' -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:8638c9979685' -drive 'file=/localdata/template/iso/CentOS-7-x86_64-Minimal-1511.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=rbd:vmStoragePool/vm-100-disk-1:mon_host=172.16.0.1;172.16.0.2;172.16.0.3:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/ceph_storage.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B6:DF:92:7A:D7:8B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1

root@prx026:~# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 15933 MB
node 0 free: 4868 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 16125 MB
node 1 free: 764 MB
node distances:
node  0  1
  0:  10  21
  1:  21  10

bootdisk: scsi0
cores: 8
hotplug: disk,network,usb,cpu
ide2: localdata:iso/CentOS-7-x86_64-Minimal-1511.iso,media=cdrom
memory: 4096
name: dmoraTest
net0: virtio=B6:DF:92:7A:D7:8B,bridge=vmbr0,tag=210
numa: 1
numa0: cpus=0;1;2;3,hostnodes=0,memory=4096,policy=bind
ostype: l26
scsi0: ceph_storage:vm-100-disk-1,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=691058c4-d0d1-4039-8b28-a256a9cb5613
sockets: 1
vcpus: 4
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back