proxmox is not using available RAM, rather uses (full) swap

[9846545]

Hello!

We are currently using a proxmox-server to virtualize about 20 windows machines (usually 6-8 hosts running concurrently).
Currently, we are experiencing an issue with RAM-Allocation.

We have 39.62% (24.91 GiB of 62.88 GiB) RAM usage and the SWAP usage is at 98.26% (12.16 GiB of 12.37 GiB)

We set the vm swapiness to 0:

root@proxmox:~# cat /proc/sys/vm/swappiness
0

Afterwards, we tried to load swap into memory using swapoff, however, after 3 days of running, one of our VMs failed to boot (stuck in BIOS), so we killed the swapoff process. The VM booted again without issues.

Practically all swapped memory is virtual machines, as indicated by this command:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less
kvm 3132980 kB
kvm 3083364 kB
kvm 1526352 kB
kvm 1459116 kB
kvm 1078276 kB
kvm 1002864 kB
kvm 953320 kB
kvm 713552 kB
kvm 621672 kB
kvm 392684 kB
pvedaemon 98136 kB
pveproxy 77640 kB
pve-ha-crm 66548 kB
snmpd 50692 k
[...]

However, all the used RAM is VMs too:

root@proxmox:~# ps aux --sort -rss
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 13590 24.4 8.2 9625248 5425908 ? Sl May27 1375:43 /usr/bin/kvm -id
root 12138 8.2 5.7 5816264 3784828 ? Sl Mar27 7682:55 /usr/bin/kvm -id
root 7970 25.7 5.0 5484724 3340648 ? Sl May27 1460:26 /usr/bin/kvm -id
root 23235 28.0 5.0 5412500 3323396 ? Sl May27 1621:32 /usr/bin/kvm -id
root 17382 6.7 5.0 5443228 3310716 ? Sl Apr03 5620:10 /usr/bin/kvm -id
root 16166 5.9 4.3 5329452 2858644 ? Sl May27 323:05 /usr/bin/kvm -id
root 24549 13.9 4.2 5345712 2792724 ? Sl May27 779:44 /usr/bin/kvm -id
root 1425 10.9 3.8 4508836 2560268 ? Sl May28 480:24 /usr/bin/kvm -id
root 2113 8.6 1.9 5629148 1287084 ? Sl Feb16 12872:35 /usr/bin/kvm -id
root 1921 0.6 0.2 3259720 137192 ? Sl Feb16 958:29 /usr/bin/kvm -id
www-data 451 0.3 0.0 567528 56480 ? S 09:36 0:04 pveproxy worker
www-data 1267 0.4 0.0 567244 56364 ? R 09:41 0:03 pveproxy worker
www-data 32160 0.1 0.0 567532 55888 ? S 09:32 0:02 pveproxy worker
www-data 1840 0.0 0.0 556652 46364 ? Ss Feb16 4:55 pveproxy

There appears to be a lot of RAM still available, however, it seems like proxmox is not using it at all!
We are not using zfs.

Additional information:

root@proxmox:~# pveversion
pve-manager/5.2-11/13c2da63 (running kernel: 4.15.18-9-pve)

 

[Hollis]

Try https://debian-administration.org/article/550/Creating_dynamic_swap_space for aditional swap

 

[9846545]

Try [Link removed because of spam filtering] for aditional swap

I am not sure that we need more swap space if there is still some RAM left for usage.
Preferrably, we would use all of the RAM, then all of the SWAP. This is why we configured the swappiness to be 0.

We already tried disabling swap, but it would just not load the content of the swap into the memory (really slow) and we ended up with a VM that would refuse to boot unless we killed the swapoff-command.

Presumably, changing the SWAP-size would only make proxmox use more of the swap and less of the RAM, which is what we do not want.

 

[adamb]

I am not sure that we need more swap space if there is still some RAM left for usage.
Preferrably, we would use all of the RAM, then all of the SWAP. This is why we configured the swappiness to be 0.

We already tried disabling swap, but it would just not load the content of the swap into the memory (really slow) and we ended up with a VM that would refuse to boot unless we killed the swapoff-command.

Presumably, changing the SWAP-size would only make proxmox use more of the swap and less of the RAM, which is what we do not want.

We have run into similar issues and we ended up disabling swap altogether. Our hosts have alot of spare ram so its not a big deal for us.

On our bigger clusters, live migrations into swap space never goes well.

 

[9846545]

We have run into similar issues and we ended up disabling swap altogether. Our hosts have alot of spare ram so its not a big deal for us.

On our bigger clusters, live migrations into swap space never goes well.

It is a production system, so rebooting is not really an option until our next scheduled maintenance period.

However, if this really is the only option left, we may be able to schedule a reboot within a week.