numa config option

[Waschbüsch]

Hi,

Since 3.4, there is this 'Enable numa' checkbox.
I am a bit confused as to what implications enabling this has?

Does it mean 'this host system is numa, therefore make use of CPU pinning, restrict memory to come from the used Core's CPU node, etc. to optimize the guest's performance' ?
Or is it rather some sort of emulation / pass through thing like 'This node should behave as if its architecture were numa'?

And what does it mean when moving VMs between nodes with different architectures?

One of my nodes is an older dual socket intel xeon system that is not numa and the other two are dual socket Opteron 6300 that are numa.

Thanks,

Martin

 

[swartz]

Hi,

Does it mean 'this host system is numa, therefore make use of CPU pinning, restrict memory to come from the used Core's CPU node, etc. to optimize the guest's performance' ?

And what does it mean when moving VMs between nodes with different architectures?

Martin

Yes, it will make use of CPU pinning (affinity). It is my understanding that enabling NUMA will change the scheduling algorithm such that it is now aware of NUMA nodes on the underlying physical CPU. Thus will attempt to keep vCPU scheduling within the same NUMA node whenever possible.

This is about ESXi but the principle applies here:
http://cloudarchitectmusings.com/20...-virtualizing-business-critical-applications/


As to what does that mean for moving VMs between nodes with different architectures... you just have to be aware of the NUMA nodes size for each CPU type you have.
If you read that article, it comes down to this:

For optimal performance, vCPU count should not exceed core count in a NUMA node. A single 8-vCPU VM might be slower then two 4-vCPU VMs.

 

[Waschbüsch]

Thanks, that is very helpful (and a good read, too).

 

[Waschbüsch]

Just activated the flag on the VMs and rebooted the machine.

numastat before:

[FONT=courier new]                           node0           node1           node2           node3[/FONT]
[FONT=courier new]numa_hit               502353939       272086894       396857088      1287887637[/FONT]
[FONT=courier new]numa_miss               85673262       344120561       362618964       496709695[/FONT]
[FONT=courier new]numa_foreign           974442412       171986626        85281381        57412063[/FONT]
[FONT=courier new]interleave_hit             28580           28604           28649           28636[/FONT]
[FONT=courier new]local_node             502181878       271884122       396666175      1287691186[/FONT]
[FONT=courier new]other_node              85845323       344323333       362809877       496906146[/FONT]

and after

[FONT=courier new]                           node0           node1           node2           node3[/FONT]
[FONT=courier new]numa_hit                  913566          701488          852956         1027058[/FONT]
[FONT=courier new]numa_miss                      0               0               0               0[/FONT]
[FONT=courier new]numa_foreign                   0               0               0               0[/FONT]
[FONT=courier new]interleave_hit             28607           28582           28662           28619[/FONT]
[FONT=courier new]local_node                892012          650560          801795          974214[/FONT]
[FONT=courier new]other_node                 21554           50928           51161           52844[/FONT]

So it seems it works perfectly. Awesome!

 

[maxprox]

Hello,
also have a look at

[h=2]Best practices for KVM on NUMA servers

[/h]http://www.admin-magazine.com/Archive/2014/20/Best-practices-for-KVM-on-NUMA-servers

regards,
maxprox

 

[vikozo]

very Interesting, but should this (NUMA) be activ or not on a Webserver KVM as example?

 

[vkhera]

very Interesting, but should this (NUMA) be activ or not on a Webserver KVM as example?

If your host has a NUMA architecture (if it has more than 2 CPU packages, it does), then you want to tell KVM that it should try to keep each VM on the same CPU package to avoid slow memory access and having to move and invalidate CPU cache lines across the processors.

 

[mjw]

Are there downsides to enabling NUMA? I'm just wondering why it's not set by default.

As this option is exposed in the GUI I think we should have a Wiki article for it. https://pve.proxmox.com/wiki/NUMA