SWAP usage 100%

[mojsiuk]

We have the same situation after upgrade 3 nodes cluster to 7.4 from 6.4 and increasing ram on node 1 with additional 128GB. Is there any solution? We change parameters in systemct to vm.swappiness=10.

 

[LnxBil]

Is there any solution?

Depends on the cause. There is enough information in this thread to analyze the problem further, yet no one provided the necessary information.

 

[r2d23k]

Hi,

Anybody know how to troubleshoot this?

 

[nicoska]

Hi all,

also I keep seeing SWAP full. Really strange:

 

[Kingneutron]

It seems to me like there is an excessive amount of swap configured. Anything above ~16GB of RAM on the host level, you don't want your server to swap much at all.


In my /etc/rc.local with zram enabled + 2GB of LVM swap (16GB host RAM):

echo 1 > /proc/sys/vm/swappiness


# Try to keep at least 100MB of free RAM at all times
echo 100000 > /proc/sys/vm/min_free_kbytes


# Default 100 - try more aggressively to reclaim inodes, etc from cache
echo 160 > /proc/sys/vm/vfs_cache_pressure

I'm also limiting ZFS ARC usage to 1573741824 bytes (~1.5GB), and have LVM + ext4 root.


Right now I have (5) VMs running and 2 containers, and (54MB) of 4GB swap usage:

swapon -s;free -h


Filename                                Type            Size            Used            Priority
/dev/dm-0                               partition       2097148         0               -2
/dev/zram0                              partition       2097148         55296           10

               total        used        free      shared  buff/cache   available
Mem:            15Gi        11Gi       2.4Gi        48Mi       1.7Gi       3.7Gi
Swap:          4.0Gi        54Mi       3.9Gi

 

[moodboom]

I'm not finding a lot of practical information in this thread. Telling people to buy TB of RAM is just passive aggressive nonsense for us poor home lab folks. But I'm not helping by ranting. Let me try to ask some specific questions. UPDATE: I apologize. I just read a little closer and there is good info, eg the post before this one by Kingneutron. Thanks and apologies!

1 If i have 128GB of total memory, how much should I typically dedicate to SWAP to run a set of VMs that consume 100% of the total memory (or more, tbh...)? Seems like something in the range of 8GB to 16GB, perhaps? But I suppose it depends on swappiness requirements, which leads to question 2.

2 I have dropped the swappiness of my VMs to 10 or lower, as we should all know that extra swapping is just a productivity killer. I want to drop the swappiness of proxmox. But we are told "any change to the system outside of the UI is dangerous and unsupported". Does the UI provide swappiness adjustments? Seems like a pretty useful basic thing to include in the UI. I can change at the command line, but again, that's against best practices.

3 Can I just set proxmox swappiness off (zero)? Are there any practical down sides to that?

Thanks all for any insights!

UPDATE: I want to try Kingneutron's suggestions. Here's what I have now, leading to much swapping, it seems:

cat /proc/sys/vm/swappiness
60
cat /proc/sys/vm/min_free_kbytes
67584

UPDATE 2: I applied Kingneutron's great suggestions, will see how it goes, thanks! I wanted to make sure changes persist, for that you need to update /etc/sysctl.conf, eg:

sudo sysctl -w vm.swappiness=1 # apply immediately but won't persist
sudo emacs /etc/sysctl.conf # and add this line: vm.swappiness = 1

 

[Kingneutron]

Try 0 instead of 1 for swappiness, later kernels changed the behavior it seems

 

[moodboom]

Thanks for all the help. Honestly, I think much of this thread content is correct: you can't easily "fix" over-allocation.

I think I just added one-too-many beefy VMs to my proxmox system and put it over the limit.

Here are my numbers...

128GB total memory with 10 running VMS, with GB mem: 32 32 3 12 24 8 6 12 32 12 = 153GB allocated

It's not magic!