Taskset CPU pinning

[dzdz]

I am running proxmox 7.1 on a Ryzen 5950x with 5 vms, and would like to pin each of the respective vms to particular cores.

After reading this post about CPU pinning, I have tried to use taskset to do so. However, when running taskset --cpu-list --all-tasks --pid 0-5 $vm-pid as suggested in the post, I receive the following error:

pid <pid> current affinity list: 0-31
pid <pid> new affinity list: 0-5
pid <pid> current affinity list: 0-31
pid <pid> new affinity list: 0-5
pid <pid> current affinity list: 0-31
taskset: failed to set pid <pid> affinity: Invalid argument

Upon inspection via ps -Tp <pid>, the pid in question has iou-wrk-<pid> threads causing taskset to fail, leaving me stumped on how to proceed.

Does anybody have any suggestions?

 

[LnxBil]

If you mind me asking: why do you want to use CPU pinning?

 

[dzdz]

@LnxBil I'm working on a high performance computing application and have noticed that the CPU scheduler seems to be negatively affecting performance. Namely, if vms are allotted cores such that the CPU is oversubscribed, performance decreases, but if the same number of vms are allotted cores such that the CPU is undersubscribed, performance increases. I'm trying to take my experiment a step further to see if CPU pegging offers further performance benefits.

 

[_gabriel]

how many cores per VM do you allocate ?
afaik, if all VMs has 32 cores, each vm wait for their run (sorry for my english)

 

[LnxBil]

@LnxBil I'm working on a high performance computing application and have noticed that the CPU scheduler seems to be negatively affecting performance. Namely, if vms are allotted cores such that the CPU is oversubscribed, performance decreases, but if the same number of vms are allotted cores such that the CPU is undersubscribed, performance increases. I'm trying to take my experiment a step further to see if CPU pegging offers further performance benefits.

In my experiments, this is not a big deal. Having the correct NUMA node (numa pinning if you will) and having hugepages available will significantly improve the performance (>10%), 1 GB hugepages especially if you have a lot of RAM (>256 GB).

 

[dzdz]

how many cores per VM do you allocate ?
afaik, if all VMs has 32 cores, each vm wait for their run (sorry for my english)

AMD 5950X has 16 cores, 32 threads. I previously assigned each of my 5 vms 8 cores, but saw performance issues. Dropping to 6 cores improved performance.

 

[dzdz]

In my experiments, this is not a big deal. Having the correct NUMA node (numa pinning if you will) and having hugepages available will significantly improve the performance (>10%), 1 GB hugepages especially if you have a lot of RAM (>256 GB).

Very interesting! Another avenue to pursue. Thank you for your input!

 

[1pawn]

Hi guys, great discussion here. My server (powerdge T630) has 2 CPUs, for a total of 32 threads. It has 256GB ram (16x16GB). That's a bit overkill for my home needs. With a couple of VM running, consumption is around 130W. I was wondering if it's possible, thanks to CPU pinning and some energy efficient management, to put the 2nd CPU to sleep, and have it to work only if needed.
For example, I wanted to use CPU pinning on some essential services (like pfsense firewall, truenas) to the CPU1, and leave some "extra" things (like gaming/windows VM with GPu pass-through) to CPU 2.
I already installed the essential PCIe cards to the PCIe slots connected to CPU1 (like network/nvme), and the "extra" ones to CPU2 (like the GPU).
I'm now missing the software magic Anybody tried this already?
If CPU pinning is not the answer, and I instead manage to find a way to temporarily put CPU2 to sleep, would Proxmox automatically run all services/VM from CPU1?
Thanks a lot!

 

[LnxBil]

With a couple of VM running, consumption is around 130W.

That is not much for a server. Try running a CPU benchmark on all CPUs for a e.g. 15 minutes and look at the numbers then.

to put the 2nd CPU to sleep, and have it to work only if needed.

You cannot put a CPU to sleep. If it does not run anything it will still manage the ram, it will go into normal idle mode and uses its normal idle power. If you want better energy efficiency, unplug a CPU or go with special low-power systems.