[SOLVED] NUMA mapping with Threadripper 2950X

Chais

New Member
Dec 1, 2018
18
1
3
I'm having trouble getting NUMA mapping to work. I'm getting the error message
Code:
kvm: -numa node,nodeid=0,cpus=4-7,cpus=20-23,memdev=ram-node0: CPU index (20) should be smaller than maxcpus (8)
Here's my topolgy:
Code:
Machine (47GB total) + Package L#0
  NUMANode L#0 (P#0 31GB)
    L3 L#0 (8192KB)
      L2 L#0 (512KB) + L1d L#0 (32KB) + L1i L#0 (64KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#16)
      L2 L#1 (512KB) + L1d L#1 (32KB) + L1i L#1 (64KB) + Core L#1
        PU L#2 (P#1)
        PU L#3 (P#17)
      L2 L#2 (512KB) + L1d L#2 (32KB) + L1i L#2 (64KB) + Core L#2
        PU L#4 (P#2)
        PU L#5 (P#18)
      L2 L#3 (512KB) + L1d L#3 (32KB) + L1i L#3 (64KB) + Core L#3
        PU L#6 (P#3)
        PU L#7 (P#19)
    L3 L#1 (8192KB)
      L2 L#4 (512KB) + L1d L#4 (32KB) + L1i L#4 (64KB) + Core L#4
        PU L#8 (P#4)
        PU L#9 (P#20)
      L2 L#5 (512KB) + L1d L#5 (32KB) + L1i L#5 (64KB) + Core L#5
        PU L#10 (P#5)
        PU L#11 (P#21)
      L2 L#6 (512KB) + L1d L#6 (32KB) + L1i L#6 (64KB) + Core L#6
        PU L#12 (P#6)
        PU L#13 (P#22)
      L2 L#7 (512KB) + L1d L#7 (32KB) + L1i L#7 (64KB) + Core L#7
        PU L#14 (P#7)
        PU L#15 (P#23)
    HostBridge L#0
      PCIBridge
        PCI 1022:43b6
          Block(Disk) L#0 "sda"
          Block(Disk) L#1 "sdd"
          Block(Disk) L#2 "sdb"
          Block(Disk) L#3 "sdc"
        PCIBridge
          PCIBridge
            PCI 8086:1539
              Net L#4 "enp5s0"
      PCIBridge
        PCIBridge
          PCIBridge
            PCI 1002:687f
      PCIBridge
        PCI 1022:7901
  NUMANode L#1 (P#1 16GB)
    L3 L#2 (8192KB)
      L2 L#8 (512KB) + L1d L#8 (32KB) + L1i L#8 (64KB) + Core L#8
        PU L#16 (P#8)
        PU L#17 (P#24)
      L2 L#9 (512KB) + L1d L#9 (32KB) + L1i L#9 (64KB) + Core L#9
        PU L#18 (P#9)
        PU L#19 (P#25)
      L2 L#10 (512KB) + L1d L#10 (32KB) + L1i L#10 (64KB) + Core L#10
        PU L#20 (P#10)
        PU L#21 (P#26)
      L2 L#11 (512KB) + L1d L#11 (32KB) + L1i L#11 (64KB) + Core L#11
        PU L#22 (P#11)
        PU L#23 (P#27)
    L3 L#3 (8192KB)
      L2 L#12 (512KB) + L1d L#12 (32KB) + L1i L#12 (64KB) + Core L#12
        PU L#24 (P#12)
        PU L#25 (P#28)
      L2 L#13 (512KB) + L1d L#13 (32KB) + L1i L#13 (64KB) + Core L#13
        PU L#26 (P#13)
        PU L#27 (P#29)
      L2 L#14 (512KB) + L1d L#14 (32KB) + L1i L#14 (64KB) + Core L#14
        PU L#28 (P#14)
        PU L#29 (P#30)
      L2 L#15 (512KB) + L1d L#15 (32KB) + L1i L#15 (64KB) + Core L#15
        PU L#30 (P#15)
        PU L#31 (P#31)
    HostBridge L#8
      PCIBridge
        PCIBridge
          PCIBridge
            PCI 1002:687f
      PCIBridge
        PCI 1022:7901
And here's my config
Code:
agent: 1
balloon: 0
bios: ovmf
boot: dc
bootdisk: scsi0
cores: 8
cpu: host
efidisk0: local-lvm:vm-100-disk-0,size=128K
hostpci0: 0b:00,x-vga=1,romfile=RVG64GO.F2,pcie=1
localtime: 1
machine: q35
memory: 12288
name: Chais-Win10
net0: virtio=02:53:BF:83:1D:EB,bridge=vmbr0
net1: virtio=FA:39:D1:ED:ED:A8,bridge=vmbr1
numa: 1
numa0: cpus=4-7;20-23,hostnodes=0,memory=12288,policy=bind
ostype: win10
sata0: none,media=cdrom
scsi0: /dev/sdb,backup=0,replicate=0,size=976762584K
scsihw: virtio-scsi-pci
sockets: 1
usb0: host=045e:0745
I don't quite understand why I'm getting this error. According to the documentation the NUMA configuration should be correct. So what am I missing here?

Edit:
When setting cores to 32 in order to be able to access all cores with the NUMA config I get the following errors:
Code:
kvm: warning: CPU(s) not present in any NUMA nodes: CPU 0 [socket-id: 0, core-id: 0, thread-id: 0], CPU 1 [socket-id: 0, core-id: 1, thread-id: 0], CPU 2 [socket-id: 0, core-id: 2, thread-id: 0], CPU 3 [socket-id: 0, core-id: 3, thread-id: 0], CPU 8 [socket-id: 0, core-id: 8, thread-id: 0], CPU 9 [socket-id: 0, core-id: 9, thread-id: 0], CPU 10 [socket-id: 0, core-id: 10, thread-id: 0], CPU 11 [socket-id: 0, core-id: 11, thread-id: 0], CPU 12 [socket-id: 0, core-id: 12, thread-id: 0], CPU 13 [socket-id: 0, core-id: 13, thread-id: 0], CPU 14 [socket-id: 0, core-id: 14, thread-id: 0], CPU 15 [socket-id: 0, core-id: 15, thread-id: 0], CPU 16 [socket-id: 0, core-id: 16, thread-id: 0], CPU 17 [socket-id: 0, core-id: 17, thread-id: 0], CPU 18 [socket-id: 0, core-id: 18, thread-id: 0], CPU 19 [socket-id: 0, core-id: 19, thread-id: 0], CPU 24 [socket-id: 0, core-id: 24, thread-id: 0], CPU 25 [socket-id: 0, core-id: 25, thread-id: 0], CPU 26 [socket-id: 0, core-id: 26, thread-id: 0], CPU 27 [socket-id: 0, core-id: 27, thread-id: 0], CPU 28 [socket-id: 0, core-id: 28, thread-id: 0], CPU 29 [socket-id: 0, core-id: 29, thread-id: 0], CPU 30 [socket-id: 0, core-id: 30, thread-id: 0], CPU 31 [socket-id: 0, core-id: 31, thread-id: 0]
kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
 
Last edited:

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
6,975
820
163
33
Vienna
numa0: cpus=4-7;20-23,hostnodes=0,memory=12288,policy=bind
i believe there is a misunderstanding

the cpus here are the guest cpus, not the host cpus
afair, qemu automatically binds the cpus to the correct numa nodes on the host
 

Chais

New Member
Dec 1, 2018
18
1
3
Ok. Then how do I expose host cores 4-7 and 20-23 to the guest as cores 0-7? And how do I make sure all memory allocated to the VM is on the same host NUMA node?
Not only do I want this for performance reasons, the VM also won't boot properly when given 16GB RAM, but works fine with 8GB. I suspect this is a result of the memory residing on different NUMA nodes.
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
6,975
820
163
33
Vienna
Then how do I expose host cores 4-7 and 20-23 to the guest as cores 0-7?
again, afair qemu does this automatically when given a hostnode

eg this config:

numa0: cpus=0-7,hostnodes=0,memory=12288,policy=bind
 
  • Like
Reactions: Chais

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!