Proxmox 4.1 vs 4.4 swap usage

[eth]

I recently reinstalled my Proxmox 4.1 servers with Proxmox 4.4. I recreated the CTs with identical configuration and software.

I've noticed a huge increase in swap usage. One of my servers actually crashed because it ran out of swap.

Here is a graph from one server that is still alive:

You can clearly see the swap usage before and after the upgrade from 4.1 to 4.4.

 

[gkovacs]

1. First of all, you decreased total swap space from 12 GB to 8 GB. Why would you do that? Swap should always be plenty...

2. Also, there is a sysctl variable that controls the aggressiveness of swapping, you can check it with the following command:

# sysctl vm.swappiness
vm.swappiness = 1

It is already set on all my nodes to 1, which means it will do the least amount of swapping possible. For virtual machine hosting, 1 is the recommended value by Red Hat, IBM, etc.

You also want to set another sysctl value to prevent running out of memory when doing certain tasks:

# sysctl vm.min_free_kbytes
vm.min_free_kbytes = 262144

The above is a good value for servers having at least 8 GB of RAM.

To make the above changes permanent, you have to add them to sysctl.conf which gets executed on boot:

echo "vm.min_free_kbytes=262144" >> /etc/sysctl.conf
echo "vm.swappiness=1" >> /etc/sysctl.conf
sysctl -w

3. Finally, is your system installed on ZFS? If yes, and your swap partition is on a zvol (rpool/swap), then you need to set a few other things, because the default Proxmox install crashes if there is too much memory being swapped out to ZFS without these settings:

zfs set primarycache=metadata rpool/swap
zfs set secondarycache=metadata rpool/swap
zfs set compression=zle rpool/swap
zfs set checksum=off rpool/swap
zfs set sync=always rpool/swap
zfs set logbias=throughput rpool/swap

You can copy and paste these into a shell, it's enough to execute them once and you won't crash more. The default Proxmox install does not set them, even though the ZFS on Linux FAQ includes them as important tunables.

You can find a few more memory tweaks in this thread:
https://forum.proxmox.com/threads/f...sts-during-high-io-on-host.30702/#post-159242

 

[joshin]

@eth: You might also want to look at limiting the ZFS ARC size. By default, it will try to use half your RAM.

 

[eth]

I don't use ZFS.

I set "vm.swappiness = 1" in sysctl.conf and did "sysctl -w -p" but it didn't help much.

My userspace uses only 61% of RAM, but still, I am 3 Gb deep in swap. And that number sometimes goes deeper to 5-6 Gb.

smem -w -p
Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%
kernel dynamic memory        36.32%     13.41%     22.91%
userspace memory             63.16%      1.73%     61.42%
free memory                   0.52%      0.52%      0.00%

Apps by swap usage:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less

proxysql 900584 kB
pve-ha-crm 49124 kB
pvedaemon 44540 kB
pve-ha-lrm 33312 kB
pve-firewall 29328 kB
pvestatd 28612 kB
lsyncd 14724 kB
haproxy 11276 kB
php 6256 kB
php 6148 kB
php 5776 kB
glusterfsd 4196 kB
searchd 4152 kB
glusterfs 4012 kB
glusterfs 3884 kB
dhcpd 3300 kB
php 2876 kB
rsyslogd 2324 kB
glusterfs 2304 kB
zabbix_server 2068 kB
zabbix_server 2068 kB
zabbix_server 2020 kB
zabbix_server 2016 kB
zabbix_server 2012 kB
zabbix_server 2012 kB
zabbix_server 2012 kB
zabbix_server 2008 kB
zabbix_server 2008 kB
zabbix_server 2004 kB

So I'm wondering why are apps swapping when I don't use much RAM in userland.

 

[Rhinox]

So I'm wondering why are apps swapping when I don't use much RAM in userland.

Maybe it's because your kernel is using the rest...


kernel dynamic memory 36.32% 13.41% 22.91%
userspace memory 63.16% 1.73% 61.42%