numa config option

Waschbüsch

Renowned Member
Dec 15, 2014
93
8
73
Munich
Hi,

Since 3.4, there is this 'Enable numa' checkbox.
I am a bit confused as to what implications enabling this has?

Does it mean 'this host system is numa, therefore make use of CPU pinning, restrict memory to come from the used Core's CPU node, etc. to optimize the guest's performance' ?
Or is it rather some sort of emulation / pass through thing like 'This node should behave as if its architecture were numa'?

And what does it mean when moving VMs between nodes with different architectures?

One of my nodes is an older dual socket intel xeon system that is not numa and the other two are dual socket Opteron 6300 that are numa.

Thanks,

Martin
 
  • Like
Reactions: rushandrush and mjw
Hi,

Does it mean 'this host system is numa, therefore make use of CPU pinning, restrict memory to come from the used Core's CPU node, etc. to optimize the guest's performance' ?

And what does it mean when moving VMs between nodes with different architectures?

Martin
Yes, it will make use of CPU pinning (affinity). It is my understanding that enabling NUMA will change the scheduling algorithm such that it is now aware of NUMA nodes on the underlying physical CPU. Thus will attempt to keep vCPU scheduling within the same NUMA node whenever possible.

This is about ESXi but the principle applies here:
http://cloudarchitectmusings.com/20...-virtualizing-business-critical-applications/


As to what does that mean for moving VMs between nodes with different architectures... you just have to be aware of the NUMA nodes size for each CPU type you have.
If you read that article, it comes down to this:

For optimal performance, vCPU count should not exceed core count in a NUMA node. A single 8-vCPU VM might be slower then two 4-vCPU VMs.
 
Last edited:
  • Like
Reactions: Omar Khalil and mjw
Just activated the flag on the VMs and rebooted the machine.

numastat before:
Code:
[FONT=courier new]                           node0           node1           node2           node3[/FONT]
[FONT=courier new]numa_hit               502353939       272086894       396857088      1287887637[/FONT]
[FONT=courier new]numa_miss               85673262       344120561       362618964       496709695[/FONT]
[FONT=courier new]numa_foreign           974442412       171986626        85281381        57412063[/FONT]
[FONT=courier new]interleave_hit             28580           28604           28649           28636[/FONT]
[FONT=courier new]local_node             502181878       271884122       396666175      1287691186[/FONT]
[FONT=courier new]other_node              85845323       344323333       362809877       496906146[/FONT]

and after

Code:
[FONT=courier new]                           node0           node1           node2           node3[/FONT]
[FONT=courier new]numa_hit                  913566          701488          852956         1027058[/FONT]
[FONT=courier new]numa_miss                      0               0               0               0[/FONT]
[FONT=courier new]numa_foreign                   0               0               0               0[/FONT]
[FONT=courier new]interleave_hit             28607           28582           28662           28619[/FONT]
[FONT=courier new]local_node                892012          650560          801795          974214[/FONT]
[FONT=courier new]other_node                 21554           50928           51161           52844[/FONT]

So it seems it works perfectly. Awesome!
 
Last edited:
  • Like
Reactions: ariunawan2105
very Interesting, but should this (NUMA) be activ or not on a Webserver KVM as example?

If your host has a NUMA architecture (if it has more than 2 CPU packages, it does), then you want to tell KVM that it should try to keep each VM on the same CPU package to avoid slow memory access and having to move and invalidate CPU cache lines across the processors.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!