Running out of RAM? VM Sluggish

[John0212]

We've recently deployed a new host with the following issue related specs:
30TB ZFS RAIDZ1 (8x 3.84TB Drives)
512GB DDR5-4800

I believe we have a RAM issue somewhere. We have 20 Windows VM's, each assigned 16GB of RAM, and occasionally a VM will hang and now allow any inputs and is extremely sluggish. During this time, our SWAP usage will shoot up to 100% of the 8GB size. Powering down a VM seems to help alleviate this issue and the impacted machine returns to normal. What I find interesting, and likely attributed to my lack of knowledge of some certain functions, is that we're only utilizing about 70% of our RAM. It never seems to go higher than this value. My understanding is that it shouldn't touch the SWAP unless we start to hit a ceiling of RAM but I could be mistaken.

I've applied these settings and have restarted the host. Anything else I should investigate? I'm a little lost at this point and not sure where to keep investigating.
VM-Swappiness = 10
ZFS_Arc_Max = 40GB

 

[Hqu]

Hey,

how are the stats when opening htop and compare the ram usage with PVEs webgui?

Best

 

[John0212]

HTOP is about 25GB RAM usage higher than PVE GUI

 

[John0212]

If it helps, I have an identical server with the only difference being less RAM (384GB) that appears to be doing the same thing. We only have 9 Windows VM's running on this one but the usage is still pretty high and using SWAP.

 

[emunt6]

Can you check on command line?

$> free -h

 

[gfngfn256]

Just my 2 cents:

I notice you're using double-socketed servers.
I don't know your M/B hardware RAM configurations - as per RAM for each socket - but maybe this is linked to sluggishness of VM's
You'll probably want to check sockets assignation as per each VM - also NUMA settings.

I don't personally use any double-socketed HW - and I know (nearly) nothing about NUMA settings.

 

[John0212]

Can you check on command line?

$> free -h

Update: I have disabled SWAP. I read this weekend that I may be in an OK position to just disable that, so I've done that for now. The change won't stick post reboot so I can roll back if needed.

 

[John0212]

Just my 2 cents:

I notice you're using double-socketed servers.
I don't know your M/B hardware RAM configurations - as per RAM for each socket - but maybe this is linked to sluggishness of VM's
You'll probably want to check sockets assignation as per each VM - also NUMA settings.

I don't personally use any double-socketed HW - and I know (nearly) nothing about NUMA settings.

Smart thinking. I've heard about NUMA being impacted by this too. These are brand new R760 servers from Dell and the RAM looks to be positioned correctly. Here is a screenshot from the iDRAC.

 

[_gabriel]

seems related to
https://forum.proxmox.com/threads/p...th-windows-server-2019-vms.130727/post-634171

try
numa_balancing=disable
as kernel boot options
try disable ksm sharing if possible
Reboot is required

 

[gfngfn256]

try
numa_balancing=disable
as kernel boot options

Maybe first try this instead:

echo 0 > /proc/sys/kernel/numa_balancing

with current boot (non persistent after reboot) - and if it works for you - add it afterwards to kernel boot options