6.2 Swap usage

[rlljorge]

Hi,

One node from my cluster are using all swap.
I don't have containers or zfs pool running.
I try defined swappiness=1 without success.

root@proxmox05:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         193219       63387         918          97      128914      128299
Swap:          8191        8190           1
root@proxmox05:~# 

top - 08:30:45 up 3 days, 13:33,  1 user,  load average: 4.96, 6.72, 6.85
Tasks: 495 total,   1 running, 494 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.5 us,  1.6 sy,  0.0 ni, 83.6 id,  8.5 wa,  0.0 hi,  0.8 si,  0.0 st
MiB Mem : 193219.9 total,    915.5 free,  63429.1 used, 128875.3 buff/cache
MiB Swap:   8192.0 total,     54.1 free,   8137.9 used. 128257.7 avail Mem

root@proxmox05:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-17
pve-kernel-4.15.18-28-pve: 4.15.18-56
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1
root@proxmox05:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-17
pve-kernel-4.15.18-28-pve: 4.15.18-56
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1
root@proxmox05:~#

I have memory, why proxmox are using all swap ?

Regards

Rodrigo

 

[t.lamprecht]

I have memory, why proxmox are using all swap ?

Maybe you had more memory use in the past, the box swapped unused stuff out but not again in after the memory situation go better (as is now) because the swapped out stuff wasn't requested since then. Just an idea.

You can use this onliner to check which processes are using swap:

for file in $(find /proc/*/status -exec grep -q 'VmSwap' {} \; -print); do awk '/^(VmSwap|Pid|Name)/{printf $2 " " $3 }END{ print ""}' $file 2>/dev/null; done | sort -k 3 -n -r | head

If you want to see if this was a unique event you could swap back everything by turning swap off temporarily and then check if swap usage increases directly afterwards:
swapoff -a && swapon -a
This may take a bit.

 

[rlljorge]

Hi,

the pid 28596 is a test machine without access running docker .... this is stranges for me

root@proxmox05:~# for file in $(find /proc/*/status -exec grep -q 'VmSwap' {} \; -print); do awk '/^(VmSwap|Pid|Name)/{printf $2 " " $3 }END{ print ""}' $file 2>/dev/null; done | sort -k 3 -n -r | head
kvm 28596 2894596 kB
kvm 20176 1516700 kB
kvm 2216 905004 kB
kvm 26262 776032 kB
kvm 11477 749896 kB
kvm 11363 599724 kB
kvm 3820 441916 kB
kvm 11668 394992 kB
kvm 2920 295528 kB
kvm 10991 241792 kB

I removed the test machine, e check swap.
<CODE>
root@proxmox05:~# free -m
total used free shared buff/cache available
Mem: 193219 54532 3625 97 135062 137152
Swap: 8191 5525 2666</CODE>

But I have anothers vms using swap ....
I checked in my monitoring system and de proxmox dashboard too and the memory usage in physical server is the same, 3 days ago a need restarted the server because this issue.

I didn't see this problem before upgrade to 6.2 and the workload is the same.

 

[t.lamprecht]

3 days ago a need restarted the server because this issue.

Why? Did the system become slow or even unusable or did you just not like the high swap usage?

swap usage itself is not necessarily bad, it can be even result in a better use of the systems memory.

the pid 28596 is a test machine without access running docker

Was this machine in use or did it just run along? As then swapping out some pages of it could be even OK.

Albeit, if it steadily increases with still a lot of free memory being around it would be strange...

 

[rlljorge]

The system vms were very slow .... because this I rebooted this servers 3 days ago. I saw all swap in use and high cpu load.

I turned off/on swap at 2 hours ago and I am monitoring the swap usage

root@proxmox05:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         193219       60346         874          99      131999      131338
Swap:          8191         613        7578
root@proxmox05:~# for file in $(find /proc/*/status -exec grep -q 'VmSwap' {} \; -print); do awk '/^(VmSwap|Pid|Name)/{printf $2 " " $3 }END{ print ""}' $file 2>/dev/null; done | sort -k 3 -n -r | head
kvm 20176 219696 kB
kvm 11363 148416 kB
kvm 11477 143204 kB
kvm 2216 138120 kB
kvm 3820 92164 kB
kvm 26262 77348 kB
kvm 11668 65012 kB
kvm 10991 37932 kB
kvm 2920 35648 kB
kvm 8777 28644 kB
root@proxmox05:~#

For now 613MB in use ... I am still monitoring

 

[LnxBil]

I am still monitoring

You have to monitor the swap in/out, not the swap usage. That's what Thomas tried to say and what is slowing down the system.

 

[creamers]

I'm having the exact same problem, enough free ram but it is all used by buff/cache.
Any suggestions ?

 

[rlljorge]

@
from the my last post 31 days ago I didn't have issue with my server again.
But all swap still in use, I still monitoring with vmstat and in/out are small.

root@proxmox05:~# uptime
 19:34:23 up 31 days, 37 min,  1 user,  load average: 4.72, 3.96, 4.58
root@proxmox05:~# free -m 
              total        used        free      shared  buff/cache   available
Mem:         193219       66810        1805         145      124603      124826
Swap:          8191        8191           0
root@proxmox05:~# vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  2 8388604 1861804 123300336 4281464    1    1   620   263    1    0 10  3 81  6  0
 3  4 8388604 1879496 123273648 4287596    0    0 12042 52837 14560 28523  8  2 83  8  0
 4  2 8388604 1875636 123268992 4294580    0    0 10644 56140 14832 29676 10  3 78 10  0
 3  2 8388604 1876664 123267728 4295860    0    0  7862  1151 12021 23378  8  2 86  5  0

 

[CRCinAU]

I just stumbled across this thread hunting for why my new proxmox nodes are using swap when there's ~40Gb of free RAM.... Running the suggestion of 'swapoff -a && swapon -a' did put things back where they were supposed to be....

ie: machine #2:


Machine #3:


Then looking at RAM usage on both, with Hour (Max) set:

Machine #2:


Machine #3:


You can see both where I ran the commands and ~4Gb got put back into RAM...

Same memory graph using "Week (maximum)" shows what I'd expect:

Machine #2:

Machine #3:


When looking at the script provided, I see:

root@cbd-pm-2:~# for file in $(find /proc/*/status -exec grep -q 'VmSwap' {} \; -print); do awk '/^(VmSwap|Pid|Name)/{printf $2 " " $3 }END{ print ""}' $file 2>/dev/null; done | sort -k 3 -n -r | head 
java 6614 398876 kB 
kvm 6493 172388 kB 
kvm 4643 157904 kB 
kvm 21048 156716 kB 
kvm 23119 144436 kB 
kvm 3156 125024 kB 
dotnet 2186 118012 kB 
kvm 32651 98660 kB 
kvm 4109 96896 kB 
dotnet1197 94096 kB

As such, all of this is either KVM or LXC processes (dotnet / java).

I wouldn't expect this system to be swapping though:

root@cbd-pm-2:~# free -m 
              total        used        free      shared  buff/cache   available 
Mem:          64071       19171       15139         779       29760       43522 
Swap:          8191           0        8191

 

[CRCinAU]

For what it's worth - as a diagnostic step, I added a file /etc/sysctl.d/swappiness.conf and added:

vm.swappiness = 0

To save a reboot, I also set this from the command line using:

sysctl vm.swappiness=0

Will be interesting to see what effect this has...

 

[caustic]

I've set swappiness to 0 but I'was running into the same problems decribed above.
I've 30 GBytes RAM for free...
Normally there are running 3 to 6 vms on one server.

Always when the backup starts we see a slight increase in swap usage.
Sometimes we see a decrease in backup performace to 2 or 3 MBytes/s (normally I see 150 - 230 MByte/s)
The backup process beginns and swap usage starts too...

After a while 5 to 10 days sometimes 20 days it is beginning to affect the vm's by decrease in performance....
Only a swappoff swappon brings it back....

I got these problems on 5 servers all have the same problem after a while they are slowing down, till the vm's are nearly unusable ...
No zfs in use only LVM containers, no special configurations....

I'm running for the moment Release 6.1.8 and 6.2.4 and 6.2.8 all with the same behavior.
Can I expect any progress if I would upgrade to current release?