100% swap usage on machine with 50% free RAM

[Underphil]

Hey all,

Running 4.4-13 as part of a 4 node cluster with a CEPH backing. On two of the hosts, I'm seeing 100% swap usage and tons of swapping even though only 226GB of 500GB of RAM is used. Other hosts show 0B swap used under similar load.

Any idea what's causing this behaviour and how to prevent it? Many thanks.

             total       used       free     shared    buffers     cached
Mem:          503G       502G       1.3G        66M         0B       274G
-/+ buffers/cache:       227G       276G
Swap:         8.0G       8.0G       360K

pve-manager/4.4-13/7ea56165 (running kernel: 4.4.59-1-pve)

 

[fabian]

potentially stuff that got swapped out in the past, but hasn't been needed (and thus swapped back in) since?

 

[manu]

swap is not necesserly bad per se, it can for instance happen on servers with very large uptime, that the kernel puts on swapspace parts of processes that are rarely used, to get more place to cache disk accesses.

What is suboptimal is when you're constantly swapping in and out, because this usually a symptom of an overloaded system.
you can check that with
vmstat 2
check the columns si/so (swapin/swapout) stay at 0 when the command is run over 30 seconds.

 

[Rhinox]

Having 500GB ram and 8GB swap does not make much sense. Swap can not be used effectively and gets filled up very easily. Either get rid of it altogether, or set up some serious size...

 

[LnxBil]

I'd suggest to install zram-config from Ubuntu to have RAM-based, compressed swap. This is a good tradeoff between turning off swap and having a too big swap space.

I hope we'll see RAM compression (and probably ram encryption) in the near future in Linux.

 

[e100]

Yes, disable disk swap and use zram.

Every Monday testing VMs would be swapped to disk after being idle all weekend and testers complain their VMs are slow. Even setting swappiness=1 would not prevent the problem.

Not had a single complaint or OOM event since removing disk swap and adding zram.

 

[dmulk]

swappiness=0 seems to have sorted this out.

<D>

 

[fireon]

I have kompletly disabled swap on all machines.