CPU Pinning with NUMA on 32 Core Epyc Cpu

[Bytales]

So my host hast a 32 core Epyc CPU. I will be creating two Gaming Windows VMs with Proxmox

1)First Stept is to set the Cpu Emulation type to Epyc. I think i know how to do this.


The Cpu is organized into 4 NUMA nodes, each node has ist own Cache, each NUMA node has 8 cores, but only two NUMA nodes have direct Memory acces.

I want to be able to pass a whole NUMA node directly to a VM. And differentiate between the ones that have direct Memory acces, and the ones that dont.

2)Is there any way to know which node can do that ? And list it so that i know which one must be assigned to a VM ?
3)Do i Need to pin the CPU/Threads belonging to a given NUMA individually, or can a pin a whole NUMA directly to a VM ?
4)Can one pin CPU/Threads/NUMAs at all to a VM ? Ive stumbled on some old post that say ProxMox 2.x doesnt Support CPU pinning ? Is that still the case with Proxmox 5.x ?

I havent found any wiki on CPU pinning !
Please advise !

 

[David Fell]

Hi - I'm in the process of learning about NUMA on my ThreadRipper. Here's a blog summarising someone else's learning: https://yunmingzhang.wordpress.com/...25660646872768084889296100104108112116120124/

I believe cpu pinning works these days, and that you want to pin the pid of the vm. numactl lets you do ranges to make groups easier to enter - there's something about cgroups I've not looked at that might be relevant here as well. taskset does the cpu pinning.

Brief - I'm learning about this stuff as well and just providing a quick note to a fellow explorer

 

[Bytales]

I found the command

numactl -H

Lists me the Core, and i can see which one are tied to the memory, thus locating the NUMA cores that have direct memory acces

So i need to pin the first line of cores to the first VM.

I found about the taskset also, but i didnt quite get the syntax. Also therese something there about somethign called PID

How do i find the VMs PID? Anybody knows ?

 

[Belokan]

I guess VM's PID is the PID of the KVM process handling a specific VM. You can get it using ps or qm:

VM ID:
root@pve3:~# ps -ef | awk '/id 101/ && ! /awk \// { print $2 }'
3798
root@pve3:~# qm list | awk '$1 == "101" { print $NF }'
3798

VM NAME:
root@pve3:~# ps -ef | awk '/name vmaccess1/ && ! /awk \// { print $2 }'
3798
root@pve3:~# qm list | awk '$2 == "vmaccess1" { print $NF }'
3798

 

[Bytales]

What is the code exactly ?

ps -ef | awk '/id 101/ && ! /awk \// { print $2 }'

and/or

 root@pve3:~# qm list | awk '$1 == "101" { print $NF }'

what is that 3798, is that the Output of the code beeing ran ? You outputed PID for your 101 Virtual machine ?

PID = Process Identification ?? (Im new to Linux)

Regardless, i think Proxmox team Needs to add the ability to pin CPU cores directly from the VM creation window. Unraid has it. It shouldn't be that hard to implement. And it is a must feature, especially for big Cluster of Servers, where you want specific vms to run on specific sets of cpu cores.

 

[Belokan]

Correct. PID stands for process ID.
And in this case, 3798 is the PID of the process (kvm) handling my 101/vmaccess1 VM.
The first command (ps +options) is used to display processes running on the host. The second (qm) is the Qemu management tool. Both outputs are passed to awk, which is a pattern manipulator, used in this case to extract a specific field based on regular expressions.

You need the PID in order to assign it to a specific processor/processor set.

 

[Bytales]

aha, the | passes Outputs to a new set of commands. Learning Linux here...

Then i would write

taskset -a -pc 0,1,2,3,4,5,6,7,32,33,34,35,36,37,38,39 3798

or would

taskset -a -pc 0-7,32-39 3798

work ?

Do i Need to run this everytime i restart the proxmox Server ?
Or does this gets saved somewhere ?

If i Need to rerun this everytime proxmox starts anew, couldnt i write this somewhere inside a file proxmox "loads" when it starts, so that it knows that VM will only run on those cores ?

 

[Bytales]

Look what i found
https://gist.github.com/ayufan/37be5c0b8fd26113a8be
I want to give
VM ID 100, Cores/Threads 0,1,2,3,4,5,6,7,32,33,34,35,16,17,18,19,20,21,22,23
VM ID 101, Cores/Threads 8,9,10,11,12,13,14,15,40,41,42,43,24,25,26,27,28,29,30,31
VM ID 102, Cores/Threads 36,37,38,39,48,49,50,51
VM ID 103, Cores/Threads 44,45,46,47,52,53,54,55

Leave for ProxMox Cores/Threads 56,57,58,59,60,61,62,63

How in gods Name do i modify that script to fit to my Needs ?

 

[Bytales]

Would be nice if someone from ProxMox "People" could join in and tell us what and how is it possible to do it with the current ProxMox Version.

 

[Bytales]

I guess VM's PID is the PID of the KVM process handling a specific VM. You can get it using ps or qm:

VM ID:
root@pve3:~# ps -ef | awk '/id 101/ && ! /awk \// { print $2 }'
3798
root@pve3:~# qm list | awk '$1 == "101" { print $NF }'
3798

VM NAME:
root@pve3:~# ps -ef | awk '/name vmaccess1/ && ! /awk \// { print $2 }'
3798
root@pve3:~# qm list | awk '$2 == "vmaccess1" { print $NF }'
3798

Just so that one may know, These commands didnt work for me, and i tried with the VM number that i have, which is 100

 

[Bytales]

isnt there something we can write in the conf file of the respective vm, something like
cpus: 0,1,2,3,4,5,6,7,32,33,34,35,16,17,18,19,20,21,22,23

I find it hard to believe a hypervisor that touts itself to be made with enteprise in mind, does not have some sort of built in cpu pinning.
And if indeed there is no native way to do it, i think it should be implemented pronto.

 

[Belokan]

The commands should work, you've missed something. What was the output ?

 

[LnxBil]

Unraid has it.

Yes, and unraid has another target market. Proxmox VE is for server virtualization.

And it is a must feature, especially for big Cluster of Servers, where you want specific vms to run on specific sets of cpu cores.

What is your use case? Virtualization is about abstraction, not running dedicated hardware and pinning it for VMs. If you want to optimise for performance inside of NUMA groups, just use the NUMA flag in your PVE configuration. I do not know of any software that acknowledges hard partitioning by KVM/Proxmox VE, do you have any examples?

 

[liquidox]

This seems like a very useful feature indeed, something I certainly would like to see in a future release and will use. (for what it's worth as non-sub)