NUMA and CPU hotplugging - best practices

[pietrek]

Hi,We have a dual socket Xeon E5-2620v2 based server running proxmox 3.4-6 with HT, 12 cores, 24 threads max, it has a NUMA enabled BIOS.I'd like to know if we should enable NUMA for each of our guest VMs (three W2012R2s running SQL Server 2008/AD controller/IIS Server) as well? Will it improve their performance, make them work in a more NUMA-aware, like they would on real hardware? Not that it's bad or anything, I'd say it's really good and we're satisfied with the setup, it's just that we'd like to do some fine tuning and make sure everything is configured optimally. I've read some docs: http://www.admin-magazine.com/Archive/2014/20/Best-practices-for-KVM-on-NUMA-servers http://www.admin-magazine.com/Archive/2014/20/Best-practices-for-KVM-on-NUMA-servers and I'd say enabling the option would be a good idea at least for linux guests. What's your opinion/experience?There's one more thing I'd like to ask you about. Should one remove/add cpu cores, or sockets when trying to make use of hotplugging? Adding sockets seems as more logical to me, since that's what a user would do with a physical server, adding cores to a CPU would be a much more complicated task, I think Am I right about this one, or is it completely irrelevant?

 

[spirit]

numa option, will enable numa inside guest and will try to map guest numa with with host numa.
So, better memory usage.

Now, with hotplug, we always plug cores and not sockets.

if you have:
sockets:2
cores:2
vcpus:1
numa:1.

you start the 1vcpu, and the cpu topology is numa0: socket0: core0,core2 , numa1:socket1: core1,core3

vcpus1 = numa0:socket0:core0

If you hotplug a second core,

vcpus:2 = numa0:socket0:core0, numa1:socket1:core1

If you hotplug a third core,

vcpus:3 = numa0:socket0:core0, numa1:socket1:core1, numa0:socket0:core3



Note that cpu hotplug is still young is qemu, qemu team is still working on this to improve it

 

[pietrek]

Thanks for the explanation.So if I want to add more CPU to an online guest the best way is to add cores instead sockets, but what's the right way to make the guest become aware of NUMA?My current setup is a Windows 2012 VM configured with one socket containing six cores and there seems to be no NUMA detected despite enabling the right option in Proxmox - coreinfo.exe returns "No NUMA nodes."Like I mentioned the host has two hexa-core Xeons, and if my assumptions are right, there have to be at least two sockets presented to the guest to make it work, yes? If so then would it be right to "partition" the VM and give it two sockets with three cores each? My opinion is yes, I'd still like to make sure.Thanks!

 

[spirit]

Thanks for the explanation.So if I want to add more CPU to an online guest the best way is to add cores instead sockets, but what's the right way to make the guest become aware of NUMA?My current setup is a Windows 2012 VM configured with one socket containing six cores and there seems to be no NUMA detected despite enabling the right option in Proxmox - coreinfo.exe returns "No NUMA nodes."Like I mentioned the host has two hexa-core Xeons, and if my assumptions are right, there have to be at least two sockets presented to the guest to make it work, yes? If so then would it be right to "partition" the VM and give it two sockets with three cores each? My opinion is yes, I'd still like to make sure.Thanks!

mmm, maybe it's a windows bug.
1socket with numa should display numa node.

But you can try 2 sockets-3cores with numa, it should work.


(BTW, try also with pve-kernel 3.10 if you can, because numa work really better with it)

 

[swartz]

Now, with hotplug, we always plug cores and not sockets.

you start the 1vcpu, and the cpu topology is numa0: socket0: core0,core2 , numa1:socket1: core1,core3

If you hotplug a second core,

vcpus:2 = numa0:socket0:core0, numa1:socket1:core1

Shouldn't it be this instead?
vcpus:2 = numa0:socket0:core0, numa0:socket0:core2
This way vCPUs are scheduled within the same NUMA group (i.e. same physical CPU in this case), thus benefiting from local-to CPU cache and avoid socket-to-socket communication.