NUMA and 4 sockets

[adamb]

Hey guys we just went live at a customer with our biggest server yet.

DL 560 with 4 sockets and 1.5TB of ram.

I believe we are running into some odd performance issues due to the way the memory is being spread out over the numa nodes.

Here is how the host looks

node 0 free: 2387 MB
node 1 free: 235 MB
node 2 free: 30654 MB
node 3 free: 80561 MB
node 4 free: 98 MB
node 5 free: 221 MB
node 6 free: 498 MB
node 7 free: 406 MB

It seems once these nodes get really low on memory, load skyrockets. Notice node 4 is pretty tight compared to the other nodes.

On the guest I have it set to 2 sockets and 71 cores and its numa output looks like this.

node 0 free: 538903 MB
node 1 free: 490631 MB

If I adjusted the VM to have more sockets would it help even this usage out any on the host?

 

[dcsapak]

If I adjusted the VM to have more sockets would it help even this usage out any on the host?

it could help, also you can configure the numa nodes by hand in the config
see man qm.conf and search for 'numa[n]'

 

[adamb]

it could help, also you can configure the numa nodes by hand in the config
see man qm.conf and search for 'numa[n]'

Appreciate the input. We are running with more sockets today to see how it goes.

 

[adamb]

Also find it interesting that the host is consuming so much more ram that the guest. Running cache=default on the disks so it does seem a bit odd to me.

Guest is using 800G of ram per the proxmox GUI (Which also roughly lines up with the usage inside the guest). Host is using 1.17TB. I wonder what the host is doing with that almost 400G of ram. Its the only VM on the host.

 

[adamb]

Pretty confident we narrowed this down to being a transparent huge page issue and our workload. Found that THP defrag was kicking on and causing high latency. So far no issues in a week with THP disabled on this monster VM with 1.5TB of ram. Things are running pretty well.

 

[dcsapak]

great to hear