SWAP usage

[Binary Bandit]

Hi All,

We recently started using the built in backup with version 7 of Proxmox. When the backup runs is pushes up swap usage.

Should I be concerned?

This is the result of the backup running through all of the VMs on this node (1 of 3) in our cluster. Well, to be clear, I'm correlating the timing of the swap being used with the backup running / this didn't happen until we started to schedule backups.




The backups start at 2AM and we typically receive a warning from Zabbix a bit after that.

There aren't any issues at present I'm just investigating as it seems unusual for swap to be used when there is about 50GB of RAM free. The above image was taken about 8AM, 4.5ish hours after the backups finished.

best,

James

 

[Dunuin]

Can't you compare swap usage vs RAM utilization graphs using zabbix? Generally swap shouldn't be used as long as there is enough free RAM.

I'm not sure how SWAP of LXCs works. Maybe the backup forces high IO load on the LXCs, the LXCs start to swap and that somehow effects the hosts SWAP too?

 

[Binary Bandit]

Here's what I see in Proxmox ... this is Max memory for a day with a dot on the time that the problem started:

Zabbix:
It looks a lot like RAM is being moved to disk for some reason.

That dip two days ago is the first time the backup ran.

 

[leesteken]

Swap getting filled (slowly) over time should not be a concern. When doing a lot of I/O, Linux can choose between using more memory for filesystem cache and swap out program memory, or use less cache and keep more programs fully in memory (controlled by vm.swappiness in /etc/sysctl.conf).
When the system has enough memory and it swaps out memory that is allocated by program but never actually touched again, it will only ever fill swap. This sounds like good use of your limited real memory as, it can be used to chace files that it in actual use instead of being idle.
If you have a lot of reading swap, then this is a sign of too little real memory or too much programs/VMs. And the lots of additional I/O will slow the system down, in addition to the swap being much slower than real memory. This will not necessarily show as a full swap, since there is also a lot of reading from it (when programs need to use swapped out memory) but this is much more troublesome.

In your case the I/O that pushed some memory out to swap was probably making the backups. The fact that that memory is still in swap, while the system has enough free memory, just signals that Linux made a good choice, otherwise it would have been needed and read back into memory. And the "free" memory contains the file system cache, which might be useful and can be reused for other purposes are any time when needed.
I would not be concerned, especially because you do not complain about performance or heavy I/O.

 

[Binary Bandit]

Thanks avw,

Another backup triggered last night and swap is holding at about 6.5 or 8GB used. There is a significantly larger amount of storage on the node that has the 6.5GB of swap usage vs other cluster nodes. My guess is that this is why SWAP is used ... based on what you are saying.

Thanks everyone for the input, I think that I'll put this one to bed ... grateful to learn a bit more about linux swap usage.