proxmox 5.0-30 free RAM SWAP full why?

C

chalan

Member
Mar 16, 2015
119
4
16
suddenly totaly slow system, IO delay 30% RAM is free, SWAP full why? how to fix it without restarting the server?

Code: 
CPU usage
13.86% of 8 CPU(s)
  
IO delay
29.92%
Load average
4.91,4.75,4.48
RAM usage
51.51% (16.18 GiB of 31.41 GiB)
  
KSM sharing
1.56 GiB
HD space(root)
15.03% (100.95 GiB of 671.65 GiB)
  
SWAP usage
100.00% (8.00 GiB of 8.00 GiB)
CPU(s)
8 x Intel(R) Atom(TM) CPU C2758 @ 2.40GHz (1 Socket)
Kernel Version
Linux 4.10.17-2-pve #1 SMP PVE 4.10.17-19 (Fri, 4 Aug 2017 13:34:37 +0200)
PVE Manager Version
pve-manager/5.0-30/5ab26bc

Code: 
root@pve-klenova:~# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0 7119012 272060 16039888 425192    1    4   748   495    0   14 44  6 44  6  0
 
Last edited:
edit: few days before i have upgraded RAM from 16GB to 32GB, is there something i have to change according swap after RAM added?
 
thank you for trying to help me... i have NO LXC containers only 4 VM's. i really dont know how to find out whats eating the swap...

root@pve-klenova:~# pidof memcached
root@pve-klenova:~# pgrep memcached

command "for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done" shows hundred of processes, im lost there...
 
Linux will generally prefer to swap out seldomly accessed sections of RAM, to proactively increase available memory for buffer or file system cache.

View memory utilisation:
Code: 
grep -i 'buff\|cach' /proc/meminfo;


We install generous amounts of RAM, so that data recently written to Ceph FileStore OSDs would be cached:
Code: 
rados bench -p rbd 120 write --no-cleanup # MBps throughput: 337/824/0  latency: 0.2s/5.1s/0.0s (avg/max/min)
rados bench -p rbd 120 rand               # MBps throughput: 1448         latency: 0.1s/1.3s/0.0s (avg/max/min)
rados bench -p rbd 120 seq                # MBps throughput: 1451         latency: 0.0s/0.7s/0.0s (avg/max/min)
rados bench -p rbd 120 write              # MBps throughput: 338/800/0  latency: 0.2s/5.9s/0.0s (avg/max/min)

Code: 
DiskSpd v2.0.17 (https://gallery.technet.microsoft.com/DiskSpd-a-robust-storage-6cd2f223)

VM specs:
  2 x Haswell-noTSX NUMA cores (host runs 2 x Intel E5-2640 v4 @ 2.40GHz with HyperThreading enabled)
  4 GB RAM (LoadReduced-DIMMs)
  Windows 2012r2 - VirtIO SCSI (vioscsi) with KRBD

Ceph Jewel FileStore hdd OSDs with SSD journals (2:1 ratio):

diskspd -b256K -d60 -h -L -o2 -t4 -r -w30 -c250M c:\io.dat
  1 read    :  379.19 MBps  1517 IOPs    write    : 163.84 MBps    655 IOPs
  2 read    :  384.15 MBps  1537 IOPs    write    : 166.14 MBps    665 IOPs
  3 read    :  371.47 MBps  1486 IOPs    write    : 160.56 MBps    642 IOPs

diskspd -b8K -d120 -h -L -o2 -t4 -r -w30 -c250M c:\io.dat
  1 read    :   30.43 MBps  3895 IOPs    write    :  13.04 MBps   1669 IOPs


diskspd -b256K -d120 -Sb -L -o2 -t4 -r -w30 -c250M c:\io.dat
  1 read    : 2652.25 MBps 10609 IOPs    write    : 1137.91 MBps  4552 IOPs
diskspd -b8K -d120 -Sb -L -o2 -t4 -r -w30 -c250M c:\io.dat
  1 read    :  764.84 MBps 97900 IOPs    write    :  328.43 MBps 42039 IOPs


Reduce the swapiness value, to reduce swap utilisation:
Code: 
    echo -e "\n# Swapping memory is 'expensive', rather don't cache\nvm.swappiness = 10" >> /etc/sysctl.conf;
    sysctl -p;
    cat /proc/sys/vm/swappiness;

No swap and almost all available memory is used as buffer or file system cache:
Code: 
top - 08:32:10 up 1 day, 14:29,  3 users,  load average: 2.66, 4.24, 4.27
Tasks: 622 total,   1 running, 621 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.0 us,  1.3 sy,  0.0 ni, 94.5 id,  2.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26405056+total, 18719416 free, 13966340+used, 10566774+buff/cache
KiB Swap: 26830438+total, 26830438+free,        0 used. 12499627+avail Mem
 
root@pve-klenova:~# grep -i 'buff\|cach' /proc/meminfo;
Buffers: 13645320 kB
Cached: 66188 kB
SwapCached: 113680 kB

what is this code you posted? i dont understand

root@pve-klenova:~# rados bench -p rbd 120 write --no-cleanup # MBps throughput: 337/824/0 latency: 0.2s/5.1s/0.0s (avg/max/min)
2017-12-06 20:06:42.348198 7f926aa2cb40 -1 did not load config file, using default settings.
no monitors specified to connect to.
couldn't connect to cluster: (2) No such file or directory

an also this?

root@pve-klenova:~# diskspd -b256K -d60 -h -L -o2 -t4 -r -w30 -c250M c:\io.dat
-bash: diskspd: command not found

i set swappines to 10 well see if it helps...
 
I am having a similar issue. 4 identically configured servers running identical tasks yet one of the servers is at just 80% of Ram usage with 50gb free yet 100% of SWAP USAGE configured at 8gb.

What precisely do I do to fix this as I am not well schooled on using command line other than how to open it.
 
You must log in or register to reply here.
Top Bottom