PCIE passthrough, GPU on numa node 1

[xrstokes]

High I've got a proxmox host up and running as my main machine at home. I also use it as my desktop by passing through a GPU to windows 10 pro. Is a dual socket xeon system and even though it has 2 16core CPU's, on my main desktop i passthrough 12 from each socket. My question is this? The gpu is connected to the second numa node, the pcie lane on the second CPU. For the best performance shouldn't the guest OS somehow be informed of this? Or might the GPU start accessing vram on the first memory bank.?

Thanks in advance.

 

[wolfgang]

Hi,
I would use NUMA with numa CPU binding.[1][2]

qm set <VMID> --numa=1 --numa<No>:cpus=<CPUID>,Hostnode=<Numanode>,policy=bind

1.)https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_virtual_machines_settings section NUMA
2.) https://pve.proxmox.com/pve-docs/pv...l_machines_with_span_class_monospaced_qm_span section numa

 

[xrstokes]

if I use a "qm set" command is the change persistent too reboots or do I need to script it?

 

[wolfgang]

It is persistent, but this change will be applied after you restart the VM.
Reboot inside the VM will not work.

 

[xrstokes]

So I'm not quite understanding the lingo a little I think what I need is something this.
qm set 100 --numa=1 --numa0:cpus=10,Hostnode=1,policy=bind,memory=12288
100 is the VMID i'm sure.
numa on is enabled I get that. Although it is already enabled in the gui.
--numa0:cpus=10 doesn't work, what I'm think i'm trying to say is that in the guest, numanode0 has 10 cpus , 12GBRAM and that these resources come from hostnode1, then I would run the same but inverted where the guest numanode1 came from hostnode0.

this is my system if it helps.
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
node 0 size: 32205 MB
node 0 free: 224 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
node 1 size: 32225 MB
node 1 free: 117 MB

Thanks in advance, almost there.

 

[wolfgang]

Please run this command to show me your Numa architecture

numactl -s

 

[xrstokes]

policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
cpubind: 0 1
nodebind: 0 1
membind: 0 1

 

[xrstokes]

BTW, wolfgang is a great forum name, check out this from my childhood.
https://www.youtube.com/watch?v=OhzlvfJZZ0s

 

[xrstokes]

qm set 100 --numa0 cpus=0-11,hostnodes=0,memory=12288,policy=bind
qm set 100 --numa1 cpus=12-23,hostnodes=1,memory=12288,policy=bind

Those 2 commands work to apply but I'm still getting a ton of numa misses. Can some one answer me this.
Does "cpus=" refer to the cpu's in the guest or the host?
Also I think there might be a bug in the parser. "cpus=0-11" works "cpus=1;2;3;4," fails with to many arguments. Am I doing it wrong?

 

[wolfgang]

You have to quote the options like

qm set 100 --numa1 "cpus=11;13,hostnodes=1,memory=12288,policy=bind"

You need the same amount of cpus in numa setting as you set at cores.

 

[xrstokes]

Thanks again mate.
Does "cpus=" refer to the cpu's in the guest or the host? Because if it's the guest I would use 0-11 as windows splits numa nodes 12 then 12. But if cpus on the host it would be 1;3;5...?

 

[wolfgang]

The cpus are the VCPU of the VM what is bind to the host numa node.