Hugepage allocation failed

Byron

Member
Apr 2, 2019
19
1
8
44
I have a machine with 128GB of RAM with an EPYC 7401P (24C 48T) and want to run 4 VM's.
Initially I had issues setting up the first machine and giving it varying amounts of RAM, but that seemed fixed when using hugepages and configuring more sockets. Setup of the template went well but now I hit another problem; when trying to start 4 equal VM's it won't start the last one (whatever machine I start last will not start). My VM's all get 10 vCPU's, 30GB of RAM and one GPU passed through.

The error I get is:
Code:
TASK ERROR: start failed: hugepage allocation failed at usr/share/perl5/PVE/QemuServer/Memory.pm line 532.

This is my grub config:
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on hugepagesz=2M hugepages=61000 default_hugepagesz=2M"

State before starting VM's:
Code:
root@ProxS01:~# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:   61000
HugePages_Free:    61000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

After starting 3 VM's looking as expected:
Code:
root@ProxS01:~# cat /proc/meminfo | grep -i huge
AnonHugePages:     45056 kB
ShmemHugePages:        0 kB
HugePages_Total:   61000
HugePages_Free:    16000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

When attempted to start the 4th VM:
Code:
root@ProxS01:~# cat /proc/meminfo | grep -i huge
AnonHugePages:     45056 kB
ShmemHugePages:        0 kB
HugePages_Total:   61409
HugePages_Free:    16409
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
I can run any combination of 3 VM's, but whichever machine is started last will fail like that, it somehow creates 400-700 extra hugepages but doesn't allocate any of the available freepages.

VM config:
Code:
balloon: 0
bootdisk: scsi0
cores: 5
cpu: host,hidden=1
hostpci0: 21:00
hugepages: 2
ide2: none,media=cdrom
memory: 30000
name: ProxS01M01
net0: virtio=CA:F9:B7:37:26:2D,bridge=vmbr0
numa: 1
ostype: l26
scsi0: local-lvm:vm-101-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=8bef3d1c-c51f-47de-ba63-f67aa343f407
sockets: 2
vmgenid: 7d1f8100-002c-420a-a85d-8253031c3a90

Giving less resources to each machine makes them work eg. 4 vCPU & 10 GB of RAM.

I have tried setting the machines as 1 socket with 10 cpu's, 2 sockets with 5 cpu's and 4 sockets with 2 - 3 cpu's, ballooning on or off, same result. I am up to date with the latest no-subscription repository.
When not using hugepages I get a timeout error like this:
Code:
TASK ERROR: start failed: command '/usr/bin/kvm -id 103 -name ProxS01M03 -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=235f9ccb-2831-4b93-9e7a-e0674e4b08db' -smp '12,sockets=4,cores=3,maxcpus=12' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/103.vnc,x509,password -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off' -m 30000 -object 'memory-backend-ram,id=ram-node0,size=7500M' -numa 'node,nodeid=0,cpus=0-2,memdev=ram-node0' -object 'memory-backend-ram,id=ram-node1,size=7500M' -numa 'node,nodeid=1,cpus=3-5,memdev=ram-node1' -object 'memory-backend-ram,id=ram-node2,size=7500M' -numa 'node,nodeid=2,cpus=6-8,memdev=ram-node2' -object 'memory-backend-ram,id=ram-node3,size=7500M' -numa 'node,nodeid=3,cpus=9-11,memdev=ram-node3' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=d7a87059-2048-4803-973a-0cfd1ae50cb3' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=43:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,multifunction=on' -device 'vfio-pci,host=43:00.1,id=hostpci0.1,bus=pci.0,addr=0x10.1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:8ff6963ed07e' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/pve/vm-103-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=5E:41:72:1B:09:14,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=pc'' failed: got timeout

I am getting a paid subscription when this machine goes into production but I'm not sold on Proxmox if I can't get my system to work.
 
Last edited:
hi,

can you post the content of

/sys/devices/system/node/node*/hugepages/hugepages-2048kB/free_hugepages
and
/sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages

for all nodes: after boot before any vm is started, and after starting each vm
 
I'm thinking this is not related to hugespages, I can start all VM's when giving them 8 vCPU's per machine and 20GB of RAM with NUMA=0, but I can't start at least one when enabling NUMA. It's a different error when not using hugepages but the same issue I think.

The error I get instead of the hugepages error (but occurs with similar circumstances)
Code:
TASK ERROR: start failed: command '/usr/bin/kvm -id 103 -name ProxS01M03 -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=235f9ccb-2831-4b93-9e7a-e0674e4b08db' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/103.vnc,x509,password -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off' -m 20000 -object 'memory-backend-ram,id=ram-node0,size=20000M' -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'vmgenid,guid=d7a87059-2048-4803-973a-0cfd1ae50cb3' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=43:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,multifunction=on' -device 'vfio-pci,host=43:00.1,id=hostpci0.1,bus=pci.0,addr=0x10.1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:8ff6963ed07e' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/pve/vm-103-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=5E:41:72:1B:09:14,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=pc'' failed: got timeout

It seems to happen as soon as I allocate more than ~20GB of RAM per machine, with or without hugepages.
 
Last edited:
I believe I found a bug in how the memory is allocated. When running numactl -H before running any VM I get this:
Code:
root@ProxS01:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32140 MB
node 0 free: 31248 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32232 MB
node 1 free: 31813 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32253 MB
node 2 free: 31922 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32252 MB
node 3 free: 32012 MB

When starting 1 VM:
Code:
root@ProxS01:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32140 MB
node 0 free: 31240 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32232 MB
node 1 free: 31808 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32253 MB
node 2 free: 2034 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32252 MB
node 3 free: 31737 MB

With 3 VM's:
Code:
root@ProxS01:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32140 MB
node 0 free: 31237 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32232 MB
node 1 free: 1729 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32253 MB
node 2 free: 2141 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32252 MB
node 3 free: 1492 MB

With 4:
Code:
root@ProxS01:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32140 MB
node 0 free: 31237 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32232 MB
node 1 free: 100 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32253 MB
node 2 free: 84 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32252 MB
node 3 free: 1549 MB

The 4th VM seems to want to take memory from the same node where VM 1 & 2 took their memory, so there is obviously not enough. This matches the earlier seen behaviour with the hugepages, but is thus not related to the hugepages.
I will try to pin the cpu's to prevent this and report back. Let me know if you want to see other info.
 
This is how it looks when starting 4 VM's with 20GB RAM each:
Code:
root@ProxS01:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32140 MB
node 0 free: 11273 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32232 MB
node 1 free: 31516 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32253 MB
node 2 free: 3751 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32252 MB
node 3 free: 207 MB

In this case the 4th machine was able to scrape just enough from node 2 & 3 which already had a VM on them, so it didn't fail. Node 1 was ignored in this instance. So the problem exists just as much but is only less likely to fail.

Can you confirm this is a bug? How can I pin the VM's to predefined nodes?
I'm trying:
Code:
balloon: 0
bootdisk: scsi0
cores: 4
cpu: EPYC,hidden=1
hostpci0: 21:00
ide2: none,media=cdrom
memory: 30000
name: ProxS01M01
net0: virtio=CA:F9:B7:37:26:2D,bridge=vmbr0
numa: 1
numa0: cpus=0;1;2;3,hostnodes=0,memory=30000,policy=preferred
ostype: l26
scsi0: local-lvm:vm-101-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=8bef3d1c-c51f-47de-ba63-f67aa343f407
sockets: 1
vmgenid: 7d1f8100-002c-420a-a85d-8253031c3a90

edit: above config worked! For others having this issue in the future: the configuration that matters is
Code:
numa0: cpus=0;1;2;3,hostnodes=0,memory=30000,policy=preferred
Like that the machine takes its cpu cores and memory from numa node 0, like that I can also optimize the most ideal PCI-e devices, optimise for memory bandwidth etc.

The automatic numa allocation is bugged on EPYC though.
 
Last edited:
Another issue found with my configuration:
When starting 4 VM's taking 30000MB from every numa (machine has 32GB of ram per numa node on an EPYC system), all seems to work fine.
When I start the 4VM's with hugepages enabled it fails every time at some point.

This snapshot is taken when the 3rd VM is attempting (and failing) to start:
Code:
root@ProxS01:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32140 MB
node 0 free: 1639 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32232 MB
node 1 free: 135 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32253 MB
node 2 free: 94 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32252 MB
node 3 free: 31336 MB

VM 1 (which runs on numa 0) acts as expected and starts within seconds, VM 2 (which runs on numa 1) takes much longer but starts eventually, however 1500MB more memory is gone in its node, VM 3 does the same and fails to start alltogether.

Edit: the same happens with giving 25GB per machine, so with 7GB overhead the machine still allocates all of the RAM within the node and fails to start:
Code:
root@ProxS01:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32140 MB
node 0 free: 6562 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32253 MB
node 1 free: 65 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32253 MB
node 2 free: 90 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32231 MB
node 3 free: 31771 MB
All machines set to 25000MB, machine 1 starts without a hitch again, 2 starts but overallocates RAM, 3 doesn't start at all.

30GB per machine without hugepages:
Code:
root@ProxS01:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32140 MB
node 0 free: 1639 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32232 MB
node 1 free: 1529 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32253 MB
node 2 free: 1613 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32252 MB
node 3 free: 1621 MB
All machines start without issue without hugepages.

TL;DR: 2 bugs for Proxmox to fix, one described here, one described in 1 post above.
 
Last edited:
Is there something I can do so I don't need to set every VM manually?
It makes automated deployment impossible, some kind of workaround would be convenient.
 
I have a ThreadRipper 2990WX w/128GB of RAM and am running into a the same basic issue. I've attempted to duplicate most of Byron's suggestions/findings w/even less success. Although we're going for different goals. I just want one vm w/most of the RAM.
Code:
root@p5pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.18-12-pve root=/dev/mapper/pve-root ro amd_iommu=on hugepagesz=2M hugepages=61000 default_hugepagesz=2M rdblacklist=mpt3sas

And the vm conf:
Code:
root@p5pve:~# cat /etc/pve/qemu-server/100.conf 
args: -cpu host,+svm
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 16
cpu: host
efidisk0: local-lvm:vm-100-disk-1,size=128K
hostpci0: 0c:00,pcie=1
hugepages: 2
ide2: local:iso/FreeNAS-11.2-U4.1.iso,media=cdrom
machine: q35
memory: 65536
name: FreeNAS11.2-U4.1
net0: virtio=EE:44:C9:BB:F2:EC,bridge=vmbr0,firewall=1
numa: 1
numa0: cpus=0;1;2;3;4;5;6;7;32;33;34;35;36;37;38;39,hostnodes=0,memory=65536,policy=preferred
onboot: 0
ostype: other
scsi0: local-lvm:vm-100-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=938d7593-854b-4160-ac1e-9df7cc658b75
sockets: 1
vga: vmware
vmgenid: 1e29ab1d-c232-4e8c-876c-bbf9de07798b

When I start the vm, it just sits and spins. Never errors out but also never completes. The only way to make it stop is to reboot the box.
Before I had added the hugepages kernel args and no numa specification, the most memory I could get it to start w/was 61GB.

It's quite vexing in the sense of wtf is the point of having so much RAM if it can't be taken advantage of...and if a hypervisor is not up to the task, well that's just plain sad.
 
I have a ThreadRipper 2990WX w/128GB of RAM and am running into a the same basic issue. I've attempted to duplicate most of Byron's suggestions/findings w/even less success. Although we're going for different goals. I just want one vm w/most of the RAM.
Code:
root@p5pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.18-12-pve root=/dev/mapper/pve-root ro amd_iommu=on hugepagesz=2M hugepages=61000 default_hugepagesz=2M rdblacklist=mpt3sas

And the vm conf:
Code:
root@p5pve:~# cat /etc/pve/qemu-server/100.conf
...

When I start the vm, it just sits and spins. Never errors out but also never completes. The only way to make it stop is to reboot the box.
Before I had added the hugepages kernel args and no numa specification, the most memory I could get it to start w/was 61GB.

It's quite vexing in the sense of wtf is the point of having so much RAM if it can't be taken advantage of...and if a hypervisor is not up to the task, well that's just plain sad.

I think you made a mistake here:
Code:
numa0: cpus=0;1;2;3;4;5;6;7;32;33;34;35;36;37;38;39,hostnodes=0,memory=65536,policy=preferred
The cpu's listed after CPU should be numbered sequentially with the core id from the VM, not the host. In other words, you should define that as:
Code:
numa0: cpus=0;1;2;3;4;5;6;7;8;9;10;11;12;13;14;15,hostnodes=0
or
Code:
numa0: cpus=0-15,hostnodes=0
Which means that numa 0 from the VM should get its cpu 0 till 15 from numa node/hostnode 0 from the host.

Also if you have 128GB of ram, you cant take 64GB per numa node as you do here, you'll be limited to ~60-62GB per Numa, so you can pass around 120-124GB.

So in your setting you should have:
Code:
cores:16
memory: 122000
numa0: cpus=0-7,hostnodes=0,memory=61000,policy=preferred
numa1: cpus=8-15,hostnodes=1,memory=61000,policy=preferred

Can you try that first and report back? This is how I work around the issue

Code:
balloon: 0
bootdisk: scsi0
cores: 40
cpu: host,hidden=1
hostpci0: 21:00
hostpci1: 22:00
hostpci2: 43:00
hostpci3: 61:00
ide2: none,media=cdrom
memory: 120000
name: ProxS01M01
net0: virtio=0E:70:09:AB:B2:C1,bridge=vmbr0
numa: 1
numa0: cpus=0-9,hostnodes=0,memory=30000,policy=bind
numa1: cpus=10-19,hostnodes=1,memory=30000,policy=bind
numa2: cpus=20-29,hostnodes=2,memory=30000,policy=bind
numa3: cpus=30-39,hostnodes=3,memory=30000,policy=bind
ostype: l26
scsi0: local-lvm:vm-101-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=42f238ae-0d94-499e-9118-ee55ef6e9252
sockets: 1
vmgenid: df0ddbac-89df-44df-b119-490e7d8caa03

Summary:
Ballooning off, hugepages off, enable numa and define what resources you are taking from every numa node. In this case I'm taking 10 cores from every numa node and 30000MB RAM. That configuration has worked without issue.

You can check what you can allocate with tools like lstopo & numactl -H. Run "numactl -H" several times while booting the VM, in my case I can see one Numa node running out of resources while the rest remain untouched if I don't use the above config.

I need to setup these VM's in production with command line though and that's where it becomes too time consuming (=not economical) to use Proxmox, so I can't use Proxmox in production and won't be getting a subscription. I like using Proxmox so it's a shame that this thread has gotten no response.

AMD EPYC is bound to take more marketshare in the datacenters, the lack of support will hurt their business.
 
Last edited:
I think you made a mistake here:
Code:
numa0: cpus=0;1;2;3;4;5;6;7;32;33;34;35;36;37;38;39,hostnodes=0,memory=65536,policy=preferred
The cpu's listed after CPU should be numbered sequentially with the core id from the VM, not the host. In other words, you should define that as:
Code:
numa0: cpus=0;1;2;3;4;5;6;7;8;9;10;11;12;13;14;15,hostnodes=0
or
Code:
numa0: cpus=0-15,hostnodes=0
Which means that numa 0 from the VM should get its cpu 0 till 15 from numa node/hostnode 0 from the host.

But given my numa topology as (after booting w/no vms ever having started):
Code:
root@p5pve:~# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
node 0 size: 64403 MB
node 0 free: 62904 MB
node 1 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
node 1 size: 0 MB
node 1 free: 0 MB
node 2 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
node 2 size: 64460 MB
node 2 free: 63527 MB
node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
node 3 size: 0 MB
node 3 free: 0 MB
node distances:
node   0   1   2   3
  0:  10  16  16  16
  1:  16  10  16  16
  2:  16  16  10  16
  3:  16  16  16  10
Other than the memory limit, wrt that node, wouldn't the cpu listing be fine? Arguably 0-7;32-39 would be easier to look at, but it should not be materially different.

Also if you have 128GB of ram, you cant take 64GB per numa node as you do here, you'll be limited to ~60-62GB per Numa, so you can pass around 120-124GB.

So in your setting you should have:
Code:
cores:16
memory: 122000
numa0: cpus=0-7,hostnodes=0,memory=61000,policy=preferred
numa1: cpus=8-15,hostnodes=1,memory=61000,policy=preferred

Can you try that first and report back? This is how I work around the issue

Code:
balloon: 0
bootdisk: scsi0
cores: 40
cpu: host,hidden=1
hostpci0: 21:00
hostpci1: 22:00
hostpci2: 43:00
hostpci3: 61:00
ide2: none,media=cdrom
memory: 120000
name: ProxS01M01
net0: virtio=0E:70:09:AB:B2:C1,bridge=vmbr0
numa: 1
numa0: cpus=0-9,hostnodes=0,memory=30000,policy=bind
numa1: cpus=10-19,hostnodes=1,memory=30000,policy=bind
numa2: cpus=20-29,hostnodes=2,memory=30000,policy=bind
numa3: cpus=30-39,hostnodes=3,memory=30000,policy=bind
ostype: l26
scsi0: local-lvm:vm-101-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=42f238ae-0d94-499e-9118-ee55ef6e9252
sockets: 1
vmgenid: df0ddbac-89df-44df-b119-490e7d8caa03

Summary:
Ballooning off, hugepages off, enable numa and define what resources you are taking from every numa node. In this case I'm taking 10 cores from every numa node and 30000MB RAM. That configuration has worked without issue.

You can check what you can allocate with tools like lstopo & numactl -H. Run "numactl -H" several times while booting the VM, in my case I can see one Numa node running out of resources while the rest remain untouched if I don't use the above config.
This is ThreadRipper so I don't have all the memory channels of EPYC. KVM complains of a missing memory argument if I have memory=0 (or anything < 1), on the numa line, or even just leaving it off.
And clearly any value results in an error about how it can't bind the memory. (because it doesn't have any)
Code:
memory: 102400
numa0: cpus=0-7;32-39,hostnodes=0,memory=51200,policy=bind
numa1: cpus=16-23;48-49,hostnodes=1,memory=0,policy=bind
numa2: cpus=8-15;40-47,hostnodes=2,memory=51200,policy=bind
numa3: cpus=24-31;56-57,hostnodes=3,memory=0,policy=bind
TASK ERROR: missing NUMA node1 memory value

Further, the way it's doing its bounds checking is wrong. It's indexing the cores from the node lines against the total # of cores. The former are ids whereas the latter is a sum.
Code:
memory: 102400
cores: 32
numa0: cpus=0-7;32-39,hostnodes=0,memory=51200,policy=bind
numa2: cpus=8-15;40-47,hostnodes=2,memory=51200,policy=bind
kvm: -numa node,nodeid=0,cpus=0-7,cpus=32-39,memdev=ram-node0: CPU index (32) should be smaller than maxcpus (32)
TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name FreeNAS11.2-U4.1 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=938d7593-854b-4160-ac1e-9df7cc658b75' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-100-disk-1' -smp '32,sockets=1,cores=32,maxcpus=32' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off' -m 102400 -object 'memory-backend-ram,id=ram-node0,size=51200M,host-nodes=0,policy=bind' -numa 'node,nodeid=0,cpus=0-7,cpus=32-39,memdev=ram-node0' -object 'memory-backend-ram,id=ram-node2,size=51200M,host-nodes=2,policy=bind' -numa 'node,nodeid=2,cpus=8-15,cpus=40-47,memdev=ram-node2' -device 'vmgenid,guid=1e29ab1d-c232-4e8c-876c-bbf9de07798b' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0c:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'vmware-svga,id=vga,bus=pcie.0,addr=0x1' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:69ea3191a724' -drive 'file=/var/lib/vz/template/iso/FreeNAS-11.2-U4.1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/pve/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=EE:44:C9:BB:F2:EC,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=q35' -cpu host,+svm' failed: exit code 1

If I bump the total cores to be > than numa node listed core-ids on the idea that I don't care, pull them from <wherever>, I get this 'loveliness':
Code:
cores: 40
memory: 62904
numa0: cpus=0-7;32-39,hostnodes=0,memory=62904,policy=bind
kvm: warning: CPU(s) not present in any NUMA nodes: CPU 8 [socket-id: 0, core-id: 8, thread-id: 0], CPU 9 [socket-id: 0, core-id: 9, thread-id: 0], CPU 10 [socket-id: 0, core-id: 10, thread-id: 0], CPU 11 [socket-id: 0, core-id: 11, thread-id: 0], CPU 12 [socket-id: 0, core-id: 12, thread-id: 0], CPU 13 [socket-id: 0, core-id: 13, thread-id: 0], CPU 14 [socket-id: 0, core-id: 14, thread-id: 0], CPU 15 [socket-id: 0, core-id: 15, thread-id: 0], CPU 16 [socket-id: 0, core-id: 16, thread-id: 0], CPU 17 [socket-id: 0, core-id: 17, thread-id: 0], CPU 18 [socket-id: 0, core-id: 18, thread-id: 0], CPU 19 [socket-id: 0, core-id: 19, thread-id: 0], CPU 20 [socket-id: 0, core-id: 20, thread-id: 0], CPU 21 [socket-id: 0, core-id: 21, thread-id: 0], CPU 22 [socket-id: 0, core-id: 22, thread-id: 0], CPU 23 [socket-id: 0, core-id: 23, thread-id: 0], CPU 24 [socket-id: 0, core-id: 24, thread-id: 0], CPU 25 [socket-id: 0, core-id: 25, thread-id: 0], CPU 26 [socket-id: 0, core-id: 26, thread-id: 0], CPU 27 [socket-id: 0, core-id: 27, thread-id: 0], CPU 28 [socket-id: 0, core-id: 28, thread-id: 0], CPU 29 [socket-id: 0, core-id: 29, thread-id: 0], CPU 30 [socket-id: 0, core-id: 30, thread-id: 0], CPU 31 [socket-id: 0, core-id: 31, thread-id: 0]
kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name FreeNAS11.2-U4.1 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=938d7593-854b-4160-ac1e-9df7cc658b75' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-100-disk-1' -smp '40,sockets=1,cores=40,maxcpus=40' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off' -m 62904 -object 'memory-backend-ram,id=ram-node0,size=62904M,host-nodes=0,policy=bind' -numa 'node,nodeid=0,cpus=0-7,cpus=32-39,memdev=ram-node0' -device 'vmgenid,guid=1e29ab1d-c232-4e8c-876c-bbf9de07798b' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0c:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'vmware-svga,id=vga,bus=pcie.0,addr=0x1' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:69ea3191a724' -drive 'file=/var/lib/vz/template/iso/FreeNAS-11.2-U4.1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/pve/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=EE:44:C9:BB:F2:EC,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=q35' -cpu host,+svm' failed: got timeout

And if I _skip_ a node, (node0, node2), it complains that there's no <skipped node> defined.
Code:
memory: 102400
cores: 16
numa0: cpus=0-7,hostnodes=0,memory=51200,policy=bind
numa2: cpus=8-15,hostnodes=2,memory=51200,policy=bind
kvm: numa: Node ID missing: 1
TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name FreeNAS11.2-U4.1 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=938d7593-854b-4160-ac1e-9df7cc658b75' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-100-disk-1' -smp '16,sockets=1,cores=16,maxcpus=16' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off' -m 102400 -object 'memory-backend-ram,id=ram-node0,size=51200M,host-nodes=0,policy=bind' -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' -object 'memory-backend-ram,id=ram-node2,size=51200M,host-nodes=2,policy=bind' -numa 'node,nodeid=2,cpus=8-15,memdev=ram-node2' -device 'vmgenid,guid=1e29ab1d-c232-4e8c-876c-bbf9de07798b' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0c:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'vmware-svga,id=vga,bus=pcie.0,addr=0x1' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:69ea3191a724' -drive 'file=/var/lib/vz/template/iso/FreeNAS-11.2-U4.1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/pve/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=EE:44:C9:BB:F2:EC,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=q35' -cpu host,+svm' failed: exit code 1

The only way I have observed that I can use more than 8 cores and > 61GB of RAM is:
Code:
numa:0
cores: 62
memory: 102400
onboot: 1
Any idea on how to specify an empty memory setting so I can define the requisite numa node lines to include the node that actually has memory?

I need to setup these VM's in production with command line though and that's where it becomes too time consuming (=not economical) to use Proxmox, so I can't use Proxmox in production and won't be getting a subscription. I like using Proxmox so it's a shame that this thread has gotten no response.

AMD EPYC is bound to take more marketshare in the datacenters, the lack of support will hurt their business.
I am feeling your pain. I'm finding myself contemplating the notion of biting the bullet as it were and ponying up for ESXI. I doubt it suffers from this kind of nonsense.

Thanks for you time/assistance btw!
 
Code:
memory: 102400

cores: 32
numa0: cpus=0-7;32-39,hostnodes=0,memory=51200,policy=bind
numa2: cpus=8-15;40-47,hostnodes=2,memory=51200,policy=bind

This fails due to what I explained earlier:
You give your VM 32 cores, you can't select/pin the cores of the host, you can only choose from what hostnode you take a certain VM core id.
So here you choose that vm core 0 till 7 + 32 till 39 are taken from hostnode 0.
Since you only give 32 cores to the VM, you can only choose 0 till 31 so hence you get the error "CPU index (32) should be smaller than maxcpus (32)" = not a bug.

In other words: you can't pin the host CPU cores, this is what you are trying but it does not work like this.

The other mistake you make here: you create 2 numa nodes on the VM, but you don't number them sequentially.

Correct would be:
Code:
memory: 102400
cores: 32
numa0: cpus=0-15,hostnodes=0,memory=51200,policy=bind
numa1: cpus=16-31,hostnodes=2,memory=51200,policy=bind

Translated:
give 32 cores to the VM, split in 2 numa nodes, take core 0 till 15 + 51200MB RAM from host-numa-node 0, take core 16 till 31 + 51200MB from host-numa-node 2.

As you understood my earlier recommendation failed due to not knowing your topology.

Try this instead:

Code:
memory: 120000
cores: 60
numa0: cpus=0-13,hostnodes=0,memory=60000,policy=preferred
numa1: cpus=14-29,hostnodes=1
numa2: cpus=30-43,hostnodes=2,memory=60000,policy=preferred
numa3: cpus=44-59,hostnodes=3

logic: take all 16 cores from host-numa 1 & host-numa 3, take 14 cores from 0 & 2 so the host has faster access to the remaining system memory. You could try it the other way around as well.

The memory parameter is optional so should work. You could add "memory=1,policy=preferred" if it nags about the missing parameter but should not be needed. (And ofc as usual make sure your memory amount adds up correctly)

The only way I have observed that I can use more than 8 cores and > 61GB of RAM is:

Code:
numa:0
cores: 62
memory: 102400
onboot: 1

...In that case I don't really see why you are going through the trouble of setting up the memory allocation manually?
 
Last edited:
So here you choose that vm core 0 till 7 + 32 till 39 are taken from hostnode 0.
Since you only give 32 cores to the VM, you can only choose 0 till 31 so hence you get the error "CPU index (32) should be smaller than maxcpus (32)" = not a bug.
In other words: you can't pin the host CPU cores, this is what you are trying but it does not work like this.
Yeah, I read your words, but they weren't in a form I could parse. Were it not for these additional data points, I would _still_ not be able to grok wtf you were trying to convey.
It was fully well not clear that the cpus on a numa nodeline were the indices used to comprise the aggregate cores number. It apeared/seemed that they were the actual cpu ids for a given node as emited from numactl -H.
e.g. That given a line like this:
Code:
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
Meant that I could _ONLY_ reference 0-7 and 32-39 from that first hostnode.

_NOW_ that that misconception is cleared up, it makes _much_ more sense. Thank you! I did try searching for this prior to posting at all. But the only hit that was most closely related to me/my issue was this thread.

Trying it your way:
Code:
args: -cpu host,+svm
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 60
cpu: host,hidden=1
efidisk0: local-lvm:vm-100-disk-1,size=128K
hostpci0: 0c:00,pcie=1
ide2: local:iso/FreeNAS-11.2-U4.1.iso,media=cdrom
machine: q35
memory: 120000
name: FreeNAS11.2-U4.1
net0: virtio=EE:44:C9:BB:F2:EC,bridge=vmbr0,firewall=1
numa: 1
numa0: cpus=0-13,hostnodes=0,memory=60000,policy=preferred
numa1: cpus=14-29,hostnodes=1
numa2: cpus=30-43,hostnodes=2,memory=60000,policy=preferred
numa3: cpus=44-59,hostnodes=3
onboot: 0
ostype: other
scsi0: local-lvm:vm-100-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=938d7593-854b-4160-ac1e-9df7cc658b75
sockets: 1
vga: vmware
vmgenid: 1e29ab1d-c232-4e8c-876c-bbf9de07798b


TASK ERROR: missing NUMA node1 memory value
Note that the same error as above occurs if I use 'memory=0,policy=preferred'.

Your way w/suggested 'memory=1,policy=preferred' addition:
Code:
args: -cpu host,+svm
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 60
cpu: host,hidden=1
efidisk0: local-lvm:vm-100-disk-1,size=128K
hostpci0: 0c:00,pcie=1
ide2: local:iso/FreeNAS-11.2-U4.1.iso,media=cdrom
machine: q35
memory: 120002
name: FreeNAS11.2-U4.1
net0: virtio=EE:44:C9:BB:F2:EC,bridge=vmbr0,firewall=1
numa: 1
numa0: cpus=0-13,hostnodes=0,memory=60000,policy=preferred
numa1: cpus=14-29,hostnodes=1,memory=1,policy=preferred
numa2: cpus=30-43,hostnodes=2,memory=60000,policy=preferred
numa3: cpus=44-59,hostnodes=3,memory=1,policy=preferred
onboot: 0
ostype: other
scsi0: local-lvm:vm-100-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=938d7593-854b-4160-ac1e-9df7cc658b75
sockets: 1
vga: vmware
vmgenid: 1e29ab1d-c232-4e8c-876c-bbf9de07798b


kvm: -object memory-backend-ram,id=ram-node1,size=1M,host-nodes=1,policy=preferred: cannot bind memory to host NUMA nodes: Invalid argument
TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name FreeNAS11.2-U4.1 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=938d7593-854b-4160-ac1e-9df7cc658b75' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-100-disk-1' -smp '60,sockets=1,cores=60,maxcpus=60' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off' -m 120002 -object 'memory-backend-ram,id=ram-node0,size=60000M,host-nodes=0,policy=preferred' -numa 'node,nodeid=0,cpus=0-13,memdev=ram-node0' -object 'memory-backend-ram,id=ram-node1,size=1M,host-nodes=1,policy=preferred' -numa 'node,nodeid=1,cpus=14-29,memdev=ram-node1' -object 'memory-backend-ram,id=ram-node2,size=60000M,host-nodes=2,policy=preferred' -numa 'node,nodeid=2,cpus=30-43,memdev=ram-node2' -object 'memory-backend-ram,id=ram-node3,size=1M,host-nodes=3,policy=preferred' -numa 'node,nodeid=3,cpus=44-59,memdev=ram-node3' -device 'vmgenid,guid=1e29ab1d-c232-4e8c-876c-bbf9de07798b' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0c:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'vmware-svga,id=vga,bus=pcie.0,addr=0x1' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:69ea3191a724' -drive 'file=/var/lib/vz/template/iso/FreeNAS-11.2-U4.1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/pve/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=EE:44:C9:BB:F2:EC,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=q35' -cpu host,+svm' failed: exit code 1


Dropping hostnodes 1 and 3, and cpu count for obvious reasons, works:
Code:
args: -cpu host,+svm
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 32
cpu: host,hidden=1
efidisk0: local-lvm:vm-100-disk-1,size=128K
hostpci0: 0c:00,pcie=1
ide2: local:iso/FreeNAS-11.2-U4.1.iso,media=cdrom
machine: q35
memory: 120000
name: FreeNAS11.2-U4.1
net0: virtio=EE:44:C9:BB:F2:EC,bridge=vmbr0,firewall=1
numa: 1
numa0: cpus=0-15,hostnodes=0,memory=60000,policy=preferred
numa1: cpus=16-31,hostnodes=2,memory=60000,policy=preferred
onboot: 0
ostype: other
scsi0: local-lvm:vm-100-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=938d7593-854b-4160-ac1e-9df7cc658b75
sockets: 1
vga: vmware
vmgenid: 1e29ab1d-c232-4e8c-876c-bbf9de07798b
It would appear that either the memory parameter is _not_ actually optional or this _is_ a bug. Given this, how to access the other 32 cores?


...In that case I don't really see why you are going through the trouble of setting up the memory allocation manually?
Because it often fails; doesn't always work.
 
Yeah, I read your words, but they weren't in a form I could parse. Were it not for these additional data points, I would _still_ not be able to grok wtf you were trying to convey.

That's what I suspected, I also found it difficult to find the correct words to describe it. :D

It would appear that either the memory parameter is _not_ actually optional or this _is_ a bug. Given this, how to access the other 32 cores?

No idea. It appears it doesn't handle Numa nodes without memory properly. This thread seems to be abandoned/ignored though so I suggest starting a new thread.

Because it often fails; doesn't always work.

My problem with the normal settings as well.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!