Swap usage on Proxmox node

[schoda]

Hello,

i've got 6 Proxmox nodes in a cluster. 5 of them do not use swap space at all. One node uses all its swap space even when enough RAM is free.

root@pm-04:~# free -m
total used free shared buff/cache available
Mem: 385591 133802 250179 404 1609 249156
Swap: 8191 6070 2121

So i run:
root@pm-04:~# find /proc -maxdepth 2 -path "/proc/[0-9]*/status" -readable -exec awk -v FS=":" '{process[$1]=$2;sub(/^[ \t]+/,"",process[$1]);} END {if(process["VmSwap"] && process["VmSwap"] != "0 kB") printf "%10s %-30s %20s\n",process["Pid"],process["Name"],process["VmSwap"]}' '{}' \;
1 systemd 268 kB
4351 systemd-journal 328 kB
18245 pmxcfs 2908 kB
18561 pvestatd 33456 kB
18562 pve-firewall 49200 kB
18694 pve-ha-crm 75164 kB
18778 pve-ha-lrm 28572 kB
20021 kvm 3121576 kB
21263 kvm 1143880 kB
23190 kvm 1148764 kB
32384 kvm 114908 kB
32416 kvm 164856 kB
34684 kvm 129356 kB
34694 kvm 549484 kB
34745 kvm 1384544 kB
50348 dsm_sa_datamgrd 11868 kB
51278 kvm 231728 kB
51370 dsm_om_connsvcd 480 kB
51371 dsm_om_connsvcd 134092 kB

To find out which processes use swap space. The fun part about those kvm processes is that the VM itself does not use any swap at all. And the RAM usage of the VM was like 2GB of 32GB. Now when i migrate one of the VMs that uses swap on the host it will use swap on the node i migrated it to.

I migrated one VM to pm-01, after migration:
root@pm-01:~# free -m
total used free shared buff/cache available
Mem: 386931 251155 126834 3256 8941 129936
Swap: 8191 443 7748

Before it was 0. Look at all the free RAM on that host.

Does anyone know why the swap is full on one node and not on others? On every host i have enough free RAM.
Does anyone know why the new process on the new host uses swap after migration?

Thanks & Cheers,
Daniel

 

[t.lamprecht]

What's your vm.swappiness sysctl vm/swappiness?

Maybe lower that to 1 or so, if you do not overcommit memory and have a bit reserves left.

Does anyone know why the swap is full on one node and not on others?

Are you using ZFS? Else, I could immagine that while now the memory usage is not high, it was at a point in the past, so the host swapped out. But as the memory usage got smaller again the swap does not automatically gets swapped in, only if those pages are requested again, so "stale" memory can be in the swap for quite a bit of time. Educated guessing, though

 

[schoda]

Hi,

all server run with vm.swappiness = 60. Memory isn't overcommited. We do not run zfs. It's not a big problem since we are going to extend the cluster anyway to get more ressources. I was just curious cause i didn't see this before.

Thanks & Cheers

 

[schoda]

Just FYI: I've read up some more about swap and vm.swappiness. I've set vm.swappiness to 1 on all of our Proxmox server.

 

[lingfish]

Might I suggest more reading, because doing this most definitely isn't doing what you think it is: https://www.howtogeek.com/449691/what-is-swapiness-on-linux-and-how-to-change-it/

 

[schoda]

I'm running with vm.swappiness 1 for over a year now. It's doing what i wanted it to do - not using that much swap on my local disks. I have enough RAM and all my VMs are located on an all flash SAN in case they want to use their swap inside the VM.