NUMA and cores per sockets + memory size

plastilin

Renowned Member
Oct 9, 2012
99
5
73
Ukraine
As far as I understand, when using multiprocessor boards based on Proxmox, you need to use NUMA in virtual machines. Are there any recommendations for enabling NUMA in virtual machines? Perhaps depending on a certain amount of RAM and the number of cores? In what types of machines is it better to use and which not?
 
@Alwin According to this documentation
If the NUMA option is used, it is recommended to set the number of sockets to the number of nodes of the host system
Now if I see my hardware
Code:
root@inc1pve17:~# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 5 6 7 24 25 26 29 30 31
node 0 size: 46958 MB
node 0 free: 19984 MB
node 1 cpus: 3 4 8 9 10 11 27 28 32 33 34 35
node 1 size: 48376 MB
node 1 free: 18819 MB
node 2 cpus: 12 13 14 18 19 20 36 37 38 42 43 44
node 2 size: 48376 MB
node 2 free: 2009 MB
node 3 cpus: 15 16 17 21 22 23 39 40 41 45 46 47
node 3 size: 48351 MB
node 3 free: 256 MB
node distances:
node   0   1   2   3
  0:  10  11  23  23
  1:  11  10  23  23
  2:  23  23  10  11
  3:  23  23  11  10

So I need to set socket Count as 4 in VM Configuration. Whether my understanding is correct??
 
Last edited:
I read the documentation. But in it I did not find whether it was necessary to enable NUMA in the advanced virtual CPU options in the settings of the virtual machine itself.

Not mandatory
Read the text below
If your system has a NUMA architecture [33] we recommend to activate the option, as this will allow proper distribution of the VM resources on the host system. This option is also required to hot-plug cores or RAM in a VM.
 
I have a first gen AMD Epyc with 32 cores and 4 NUMA nodes, i.e. 8 cores per Node on the same socket. How can I make sure proxmox deploys VMs on cores of the same node? Is it correct to choose "Enable NUMA", 4 sockets and 2 cores to get 8 cores on the same node?
 
I also try to understand the relationship between NUMA hosts and VMs. Let's have a practical example: Dual-CPU Server with 8 cores / 16 threads each, resulting in 32 cores shown in Proxmox. And let's assume 128 GB RAM divided in 64 GB for each CPU.

What would be the best settings for different VM sizes?

1 vCore VM - i guess enabling NUMA makes no difference?

2, 4 or 8 vCore VM - enabling NUMA? 1 or 2 sockets?

10 or more vCore VM - my understanding is, that enabling NUMA is a must here, as well as assigning 2 sockets.

Is there a performance gain when assigning more than 16 vCores to a VM, since the system only has 16 physical cores total.

Does it make any sense to assign an odd amount of vCores (3, 5, 7, etc.)?

What about RAM? My understanding is, that I should enable NUMA if I assign more than 64 GB RAM to a VM. (More RAM then a single NUMA core has)

I am assuming Windows 10 or Linux (recent kernels) guests only.
 
There are some simple rules for VM sizing.
1. The host needs some room to breathe and schedule VMs. Make your largest VM not bigger than what you have on physical minus at least 2 cores. Depending on what storage you use (NFS, iscsi) or what othwr service are running on the host that value (2) might be higher. Threads don't count!
2. A VM must not span numa nodes (memory and cores) to provide best performance. So if your numa node is 8C and 64 GB RAM then this is your biggest VM unless...
3. ...You use numa. In such a case I would say it doesn't matter if you replicate your numa architecture or not. It is just important that 1. and 2. are not violated. You need to think things through...
4. Size as small as possible. All vcpus mean an overhead!
5. Odd numbers of cores are OK. It violates our (IT) feelings because we are thinking in "base 2" terms but in the end it doesn't matter. There have been 3C processors a long time ago and they worked just fine.

HTH
 
  • Like
Reactions: masgo

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!