Best practices for VM NUMA and socket configuration

[fossaaen]

Hi all,

We have a question regarding best practices and recommended settings for NUMA and the number of CPU sockets assigned to a VM.

We need to support memory hot-plug, which means we are currently running these VMs with NUMA=1. Our hypervisors typically have 2–4 CPU sockets, and for many years we have created VMs with the same number of virtual sockets as the underlying hypervisor, while adjusting the number of vCPUs according to the VM's requirements.

Is this the recommended approach, or would it generally be better to use NUMA=1, SOCKETS=1, and simply increase the number of vCPUs as required? Then, only increase the number of virtual sockets when the VM requires more CPU or memory than can reasonably be provided by a single NUMA node/socket.

The reason we are asking is that, from time to time, we see an issue on some Ubuntu VMs where nfs-kernel-serverfails to restart after an unattended upgrade, with an error such as:

rpc.nfsd[1212027]: error starting threads: errno 12 (Cannot allocate memory)
The VM has sufficient memory configured, so at first glance this does not appear to be a memory capacity issue. When debugging the problem, however, it appears that the kernel has difficulty allocating sufficiently large contiguous memory blocks (higher-order allocations) from Node 0.

Our understanding is that kernel allocations are subject to NUMA locality and, depending on the allocation, may require memory from the local NUMA node, unlike ordinary user-space processes which can generally allocate memory from other nodes when allowed by their NUMA policy.

Looking at /proc/buddyinfo, we typically see that, immediately after the failure, the higher-order contiguous memory on one of the NUMA nodes is very low or unavailable. This leads us to suspect that memory fragmentation may be the underlying cause.

During an apt update/apt upgrade, memory usage changes significantly and may result in increased fragmentation. The higher-order blocks required by nfsd are then temporarily unavailable, causing the restart to fail. If we wait a minute or two and try again, the kernel appears to have had time to compact/defragment memory, and nfs-kernel-server starts successfully. The issue then typically does not occur again for another couple of weeks.

For example, the following is typical /proc/buddyinfo output immediately after a failure:

Node 0, zone DMA 25 12 5 3 8 7 5 2 1 2 0
Node 0, zone DMA32 207 28 26 103 78 27 20 6 3 5 4
Node 0, zone Normal 409 733 137 94 148 67 23 104 42 1 0
Node 1, zone DMA32 4312 2915 2234 1623 860 332 86 16 11 6 6
Node 1, zone Normal 2132 1029 1034 1126 497 315 226 57 14 7 7
As can be seen, the distribution of available blocks between Node 0 and Node 1 is quite uneven. This is what led us to our current theory.

Does this sound like a plausible explanation for the nfsd allocation failure? And, more importantly, could our VM socket/NUMA configuration be contributing to the problem?

Any thoughts or recommendations regarding the preferred NUMA/socket configuration for VMs would be appreciated.

 

[jsterr]

Is this the recommended approach, or would it generally be better to use NUMA=1, SOCKETS=1, and simply increase the number of vCPUs as required? Then, only increase the number of virtual sockets when the VM requires more CPU or memory than can reasonably be provided by a single NUMA node/socket.

According to the docs: "If the NUMA option is used, it is recommended to set the number of sockets tothe number of nodes of the host system." Have you tried that and checked how it changes your situation/problem?

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_numa

During an apt update/apt upgrade, memory usage changes significantly and may result in increased fragmentation. The higher-order blocks required by nfsd are then temporarily unavailable, causing the restart to fail. If we wait a minute or two and try again, the kernel appears to have had time to compact/defragment memory, and nfs-kernel-server starts successfully. The issue then typically does not occur again for another couple of weeks.

I cant help here but wanted to point out that you should never user apt upgrade on a pve-system, as it can break the system because of how dependencys are done on apt upgrade. use apt dist-upgrade / apt full-upgrade or pveupdate/pveupgrade.

 

[fossaaen]

According to the docs: "If the NUMA option is used, it is recommended to set the number of sockets tothe number of nodes of the host system." Have you tried that and checked how it changes your situation/problem?

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_numa



I cant help here but wanted to point out that you should never user apt upgrade on a pve-system, as it can break the system because of how dependencys are done on apt upgrade. use apt dist-upgrade / apt full-upgrade or pveupdate/pveupgrade.

Thank you for your reply.
Just to clarify, its not apt upgrade on pve's that is struggling, its the apt upgrades with unattended upgrades inside the ubuntu VMs.

Yes,this is the setup we have used for many years (If the NUMA option is used, it is recommended to set the number of sockets tothe number of nodes of the host system), but lately seeing that problem with uneven load between nodes, and that kernel processes must get its resources from same node as started on, lead us to check if we could avoid the problem if we ran with 1 socket, thus having all the resources available also for kernel processes, if that makes sense ?

 

[cwt]

I think the recommendation to match the number of VM sockets to the number of host NUMA nodes is a bit too easy to misinterpret.

Personally, I would use 1 socket as the default and only expose multiple vNUMA nodes to a guest if the VM is actually large enough to benefit from NUMA awareness.

For example, if a VM with 16 vCPUs and 64 GB RAM can comfortably fit into a single physical NUMA node, I don’t see much benefit in presenting 2 or 4 NUMA nodes to the guest just because the host has that topology. It only gives the guest OS additional NUMA boundaries it now has to manage.

For larger VMs which necessarily span multiple physical NUMA nodes, this is obviously different. In that case I would enable NUMA and try to create a sensible and symmetric vNUMA topology matching the resources the VM is expected to consume.

So my rule of thumb would rather be:

- small/medium VM fitting into one host NUMA node: 1 socket, usually no vNUMA
- large VM spanning host NUMA nodes: enable vNUMA and size the nodes deliberately
- don’t use the host NUMA node count as the VM socket count by default

Regarding your nfsd issue: the buddyinfo output is actually a good example of why exposing NUMA unnecessarily can be counterproductive. Node 0 is heavily fragmented/depleted in the higher orders while Node 1 still has plenty available. From the guest’s point of view these are separate NUMA memory domains, although the VM might not have needed that distinction in the first place.

One important question remains, though: how well does Proxmox/QEMU actually guarantee that the guest vNUMA topology corresponds to physical host NUMA placement if no explicit CPU and memory binding is configured?

Without that binding, simply configuring e.g. 2 sockets / 2 vNUMA nodes in the VM does not necessarily mean that vNUMA node 0 is backed by host NUMA node 0 and vNUMA node 1 by host NUMA node 1.

That distinction is probably more important than the socket count itself.