swap usage

[PaulVM]

I noticied that the 4.4 version seems to use all the available swap space.
In my 1.x, 2.x, 3.x installations I never had this problem. I can solve it with a "swapoff -a && swapon -a", but I can't understad why this happen and how I can solve it.
The host phisycal memory is 32 GB. The total VMs memory allocated is less than 16 GB.
The server have the same config and are all standard install without ZFS, without lvm-thin and with software RAID-1 from OVH template:

# df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 6.3G 345M 6.0G 6% /run
/dev/md2 923G 488G 389G 56% /
tmpfs 16G 43M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/pve-data 906G 143G 718G 17% /var/lib/vz
/dev/fuse 30M 20K 30M 1% /etc/pve

Here the memory usage:

root@pmx4a:/etc/pve/lxc# free
total used free shared buffers cached
Mem: 32936996 32433844 503152 134844 14950204 3168440
-/+ buffers/cache: 14315200 18621796
Swap: 2095096 2031424 63672
root@pmx4a:/etc/pve/lxc# swapoff -a && swapon -a
root@pmx4a:/etc/pve/lxc# free
total used free shared buffers cached
Mem: 32936996 32710736 226260 409632 13513588 3175924
-/+ buffers/cache: 16021224 16915772
Swap: 2095096 1560 2093536

root@pmx4b:/etc/postfix# free
total used free shared buffers cached
Mem: 32937172 32685636 251536 494016 6521660 2008824 -/+ buffers/cache: 24155152 8782020
Swap: 2095096 2094996 100
root@pmx4b:/etc/postfix# swapoff -a && swapon -a
root@pmx4b:/etc/postfix# free
total used free shared buffers cached
Mem: 32937172 32685908 251264 685264 5187300 2046684 -/+ buffers/cache: 25451924 7485248
Swap: 2095096 144 2094952

He need to "swapoff -a && swapon -a" once or twice for week, while 3.4 install (24/32 GB allocated) as usually don't have this beauvoir:

root@pmx34~# free total used free shared buffers cached
Mem: 32934468 32659672 274796 0 4134192 17794100
-/+ buffers/cache: 10731380 22203088
Swap: 2095096 18692 2076404
root@pmx34:~# w
12:15:26 up 113 days, 52 min, 9 users, load average: 0.41, 0.28, 0.28

Same for all my remaining 3.x installations and in the past for the 2.x and 1.x.

Any hints?

Regards, P.

 

[ghusson]

I don't have this problem. Look at swapiness kernel configuration, and try to play with it ?

 

[fireon]

We use a very long time swapiness=0 on every server. I thing this should be the default from Proxmox. A virutalsation should not be swapped, only on emergency.

 

[ghusson]

Put swapiness=1 and reboot
swapiness=0 has lead to OOM kills from what I have seen on the web
Maybe you have an other side effect ?

What disk mode do you use on your VMs (ex : default (no cache)) ?

 

[fireon]

Never had OOM kills with that, use this swapiniss=0 on 40+ Servers. Only with this one kernel... i think it was 4.4.35-1. but this was an kernelbug.
default (no cache): that we use always, because HWraid have his own cache. And ZFS and Ceph too.

 

[PaulVM]

swappiness default is 60 both for 3.4 and 4.4.
I But the different kernel seems use a very different swap policy.
I tried to set swappiness to 10 into 3 different 4.4 servers (and cleared swap). In a couple of days we can see if there is any differenty result

Best Regards, P

 

[ghusson]

I have no other idea. Maybe try to trace which programs swaps ?
One howto found here :
https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/

 

[PaulVM]

trace which programs is using swap isn't a simple activity :-(

Anyway, done some tests:
Setting swappiness to 10 seems give no differences.
Setting swappiness to 1 seems to slow down the swap consumption, but the trend seems similar.

I have read different opinion to the option of setting swappiness to 0.
Anyone have tried?

Thanks, P.

 

[fireon]

I have read different opinion to the option of setting swappiness to 0.
Anyone have tried?

The server will only swap if no memory can allocate any more.

 

[PaulVM]

Tried swappiness to 0 into 2 server.
Similar behavioural of swappiness to 1
Setted the value via sysctl vm.swappiness=0 without reboot the server.
box1 pveversion report 4.3-12 kernel 4.4.24-1-pve
box2 pveversion report 4.4-5 kernel 4.4.35-2-pve

For both: The host phisycal memory is 32 GB. The total VMs memory allocated is less than 16 GB.

Best Regards, P.

 

[eth]

For both: The host phisycal memory is 32 GB. The total VMs memory allocated is less than 16 GB.

I'm using latest proxmox. My userspace uses only 61% of RAM, but still, I am 3 Gb deep in swap. And that number sometimes goes deeper to 5-6 Gb.

smem -w -p
Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%
kernel dynamic memory        36.32%     13.41%     22.91%
userspace memory             63.16%      1.73%     61.42%
free memory                   0.52%      0.52%      0.00%

Apps by swap usage:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less

proxysql 900584 kB
pve-ha-crm 49124 kB
pvedaemon 44540 kB
pve-ha-lrm 33312 kB
pve-firewall 29328 kB
pvestatd 28612 kB
lsyncd 14724 kB
haproxy 11276 kB
php 6256 kB
php 6148 kB
php 5776 kB
glusterfsd 4196 kB
searchd 4152 kB
glusterfs 4012 kB
glusterfs 3884 kB
dhcpd 3300 kB
php 2876 kB
rsyslogd 2324 kB
glusterfs 2304 kB
zabbix_server 2068 kB
zabbix_server 2068 kB
zabbix_server 2020 kB
zabbix_server 2016 kB
zabbix_server 2012 kB
zabbix_server 2012 kB
zabbix_server 2012 kB
zabbix_server 2008 kB
zabbix_server 2008 kB
zabbix_server 2004 kB

 

[PaulVM]

Done a "swapoff -a && swapon -a" 24 h ago.

# free
             total       used       free     shared    buffers     cached
Mem:      32936996   32037668     899328     709452   12709024    3805844
-/+ buffers/cache:   15522800   17414196
Swap:      2095096    1539740     555356

smem -w -p
Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%
kernel dynamic memory        87.98%     86.54%      1.44%
userspace memory              9.42%      0.51%      8.90%
free memory                   2.60%      2.60%      0.00%

# cat /proc/sys/vm/swappiness
0

# pveversion
pve-manager/4.4-5/c43015a5 (running kernel: 4.4.35-2-pve)

4 LXC container, no KVM. 1+2+2+6 GB Memory for the Containers (assigned resources, less really used)
The summary screen for this server reports::
RAM usage 47.73% (14.99 GiB of 31.41 GiB)
SWAP usage 72.91% (1.46 GiB of 2.00 GiB)

In few hours it wiil be at 100% ... :-(
But I don't understand why.

Really annoying situation ... :-(

 

[ghusson]

I don't use LXC containers. I think that is the point. From here : https://forum.proxmox.com/threads/how-is-swap-handled-by-lxc.26756/
I understand that SWAP in LXC use SWAP on host. Thus, it can be a swapiness problem, at LXC level. See : https://lists.linuxcontainers.org/pipermail/lxc-users/2010-August/000814.html

 

[mailinglists]

My LXCs slowly consume more and more swap, even when there is plenty of free RAM.
Up to a point where LXC guest consumes more swap than it's available.

I'm guessing there is some sort of memory leak.. I don't know.
Will upgrade the whole install soon and check if the problem persists.
Then I might do a bug report.

[root@www ~]# free -h
total used free shared buff/cache available
Mem: 33G 6.9G 25G 1.3G 879M 25G
Segmentation fault
[root@www ~]# free -m
total used free shared buff/cache available
Mem: 34000 7118 26002 1357 879 26002
Swap: 512 514 18014398509481982
[root@www ~]# free
total used free shared buff/cache available
Mem: 34816000 7289024 26626032 1389684 900944 26626032
Swap: 524288 526592 -2304

Attached graph shows slow consumption of last megabytes swap.


Try and see, if you can also get negative SWAP numbers, if you just wait. Then we have the same problem.

 

[PaulVM]

Setting vm.swappiness = 0 in /etc/sysctl.conf + update Proxmox and reboot seems to solve my swap problems (the simple change of swappiness without reboot (and update), give no results).

Anyway, I still have strange behavioural in all my 4.4 boxes: high load (compared with 3.4), and strange swap peak in the VMs:

Mon Apr  3 19:30:01 CEST 2017
              total        used        free      shared  buff/cache   available
Mem:        6324224     4138704       69376      508656     2116144       69376
Swap:       1048576           0     1048576
Mon Apr  3 19:30:01 CEST 2017
              total        used        free      shared  buff/cache   available
Mem:        6324224     4145328       62772      508768     2116124       62772
Swap:       1048576     1048576           0
Mon Apr  3 19:30:06 CEST 2017
              total        used        free      shared  buff/cache   available
Mem:        6324224     4152424       55684      508728     2116116       55684
Swap:       1048576           0     1048576
Mon Apr  3 19:30:11 CEST 2017
              total        used        free      shared  buff/cache   available
Mem:        6324224     4153972       54140      508724     2116112       54140
Swap:       1048576           0     1048576
Mon Apr  3 19:30:16 CEST 2017
              total        used        free      shared  buff/cache   available
Mem:        6324224     4134444       73672      508688     2116108       73672
Swap:       1048576           0     1048576

randomly the system reports 100% swap usage for some istants. You can see swap=0 at 19:30:01; swap = 100 less that a second later and swap = 0 five second later ...
Tried to monitor to understand what is generating this or because this happen without success (but my knowledge of the deep things of Proxmox is very limited).
If someone more expert can give us some info or explain what we are doing wrong ... ;-)
Regards, P.

 

[mailinglists]

LOL, after update to 4.4-13/7ea56165, i see exactly the same with my setup. SWAP sometimes drops to 0%. Here is how it looks when monitoring checks for it at that exact moment.