As far as I understand, when using multiprocessor boards based on Proxmox, you need to use NUMA in virtual machines. Are there any recommendations for enabling NUMA in virtual machines? Perhaps depending on a certain amount of RAM and the number of cores? In what types of machines is it better to use and which not?
@Alwin According to this documentation
If the NUMA option is used, it is recommended to set the number of sockets to the number of nodes of the host system
Now if I see my hardware
So I need to set socket Count as 4 in VM Configuration. Whether my understanding is correct??
If the NUMA option is used, it is recommended to set the number of sockets to the number of nodes of the host system
Now if I see my hardware
Code:
root@inc1pve17:~# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 5 6 7 24 25 26 29 30 31
node 0 size: 46958 MB
node 0 free: 19984 MB
node 1 cpus: 3 4 8 9 10 11 27 28 32 33 34 35
node 1 size: 48376 MB
node 1 free: 18819 MB
node 2 cpus: 12 13 14 18 19 20 36 37 38 42 43 44
node 2 size: 48376 MB
node 2 free: 2009 MB
node 3 cpus: 15 16 17 21 22 23 39 40 41 45 46 47
node 3 size: 48351 MB
node 3 free: 256 MB
node distances:
node 0 1 2 3
0: 10 11 23 23
1: 11 10 23 23
2: 23 23 10 11
3: 23 23 11 10So I need to set socket Count as 4 in VM Configuration. Whether my understanding is correct??
Last edited:
I read the documentation. But in it I did not find whether it was necessary to enable NUMA in the advanced virtual CPU options in the settings of the virtual machine itself.
Not mandatory
Read the text below
If your system has a NUMA architecture [33] we recommend to activate the option, as this will allow proper distribution of the VM resources on the host system. This option is also required to hot-plug cores or RAM in a VM.
This option is to enable numa on vm, which is recommended when you have numa enabled hardware, it is not mandatory. If you have numa hardware, it will give you better performance if you use it
I have a first gen AMD Epyc with 32 cores and 4 NUMA nodes, i.e. 8 cores per Node on the same socket. How can I make sure proxmox deploys VMs on cores of the same node? Is it correct to choose "Enable NUMA", 4 sockets and 2 cores to get 8 cores on the same node?
I also try to understand the relationship between NUMA hosts and VMs. Let's have a practical example: Dual-CPU Server with 8 cores / 16 threads each, resulting in 32 cores shown in Proxmox. And let's assume 128 GB RAM divided in 64 GB for each CPU.
What would be the best settings for different VM sizes?
1 vCore VM - i guess enabling NUMA makes no difference?
2, 4 or 8 vCore VM - enabling NUMA? 1 or 2 sockets?
10 or more vCore VM - my understanding is, that enabling NUMA is a must here, as well as assigning 2 sockets.
Is there a performance gain when assigning more than 16 vCores to a VM, since the system only has 16 physical cores total.
Does it make any sense to assign an odd amount of vCores (3, 5, 7, etc.)?
What about RAM? My understanding is, that I should enable NUMA if I assign more than 64 GB RAM to a VM. (More RAM then a single NUMA core has)
I am assuming Windows 10 or Linux (recent kernels) guests only.
What would be the best settings for different VM sizes?
1 vCore VM - i guess enabling NUMA makes no difference?
2, 4 or 8 vCore VM - enabling NUMA? 1 or 2 sockets?
10 or more vCore VM - my understanding is, that enabling NUMA is a must here, as well as assigning 2 sockets.
Is there a performance gain when assigning more than 16 vCores to a VM, since the system only has 16 physical cores total.
Does it make any sense to assign an odd amount of vCores (3, 5, 7, etc.)?
What about RAM? My understanding is, that I should enable NUMA if I assign more than 64 GB RAM to a VM. (More RAM then a single NUMA core has)
I am assuming Windows 10 or Linux (recent kernels) guests only.
There are some simple rules for VM sizing.
1. The host needs some room to breathe and schedule VMs. Make your largest VM not bigger than what you have on physical minus at least 2 cores. Depending on what storage you use (NFS, iscsi) or what othwr service are running on the host that value (2) might be higher. Threads don't count!
2. A VM must not span numa nodes (memory and cores) to provide best performance. So if your numa node is 8C and 64 GB RAM then this is your biggest VM unless...
3. ...You use numa. In such a case I would say it doesn't matter if you replicate your numa architecture or not. It is just important that 1. and 2. are not violated. You need to think things through...
4. Size as small as possible. All vcpus mean an overhead!
5. Odd numbers of cores are OK. It violates our (IT) feelings because we are thinking in "base 2" terms but in the end it doesn't matter. There have been 3C processors a long time ago and they worked just fine.
HTH
1. The host needs some room to breathe and schedule VMs. Make your largest VM not bigger than what you have on physical minus at least 2 cores. Depending on what storage you use (NFS, iscsi) or what othwr service are running on the host that value (2) might be higher. Threads don't count!
2. A VM must not span numa nodes (memory and cores) to provide best performance. So if your numa node is 8C and 64 GB RAM then this is your biggest VM unless...
3. ...You use numa. In such a case I would say it doesn't matter if you replicate your numa architecture or not. It is just important that 1. and 2. are not violated. You need to think things through...
4. Size as small as possible. All vcpus mean an overhead!
5. Odd numbers of cores are OK. It violates our (IT) feelings because we are thinking in "base 2" terms but in the end it doesn't matter. There have been 3C processors a long time ago and they worked just fine.
HTH
