NUMA and 4 sockets

adamb

Famous Member
Mar 1, 2012
1,322
72
113
Hey guys we just went live at a customer with our biggest server yet.

DL 560 with 4 sockets and 1.5TB of ram.

I believe we are running into some odd performance issues due to the way the memory is being spread out over the numa nodes.

Here is how the host looks

node 0 free: 2387 MB
node 1 free: 235 MB
node 2 free: 30654 MB
node 3 free: 80561 MB
node 4 free: 98 MB
node 5 free: 221 MB
node 6 free: 498 MB
node 7 free: 406 MB

It seems once these nodes get really low on memory, load skyrockets. Notice node 4 is pretty tight compared to the other nodes.

On the guest I have it set to 2 sockets and 71 cores and its numa output looks like this.

node 0 free: 538903 MB
node 1 free: 490631 MB

If I adjusted the VM to have more sockets would it help even this usage out any on the host?
 
If I adjusted the VM to have more sockets would it help even this usage out any on the host?
it could help, also you can configure the numa nodes by hand in the config
see man qm.conf and search for 'numa[n]'
 
Also find it interesting that the host is consuming so much more ram that the guest. Running cache=default on the disks so it does seem a bit odd to me.

Guest is using 800G of ram per the proxmox GUI (Which also roughly lines up with the usage inside the guest). Host is using 1.17TB. I wonder what the host is doing with that almost 400G of ram. Its the only VM on the host.
 
Pretty confident we narrowed this down to being a transparent huge page issue and our workload. Found that THP defrag was kicking on and causing high latency. So far no issues in a week with THP disabled on this monster VM with 1.5TB of ram. Things are running pretty well.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!