High SWAP usage

[TechLineX]

Hello,

actually, I´ve the following problem:

- I´m running several LXC-Containers over different hosts in a cluster
- Every LXC-Container has no "own" swap
- But the hosts swap is at 100% (8/8G ram)
- If the swap is at 100%, the whole system gets so slow (migrating, booting, moving, ..)

What is to do? Is it the wrong way to set up the lxc-containers with zero swap?

Best regards

 

[Faris Raouf]

How free RAM is there on the system?

With that in mind, I cannot answer your question directly, but I can tell you that I have encountered a similar issue: In a container with OR WITHOUT swap, it is possible for applications running within the Container to cause 100% swap use on the Node, while no Swap (if configured) is used in the Container.

I resolved the problem by changing the way the application I was running in the Container worked. Now it uses no swap.

You can find out what is using swap in a number of ways. There are some suggestions here: https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/ (scroll down to Listing all processes using swap space)

 

[TechLineX]

How free RAM is there on the system?

With that in mind, I cannot answer your question directly, but I can tell you that I have encountered a similar issue: In a container with OR WITHOUT swap, it is possible for applications running within the Container to cause 100% swap use on the Node, while no Swap (if configured) is used in the Container.

I resolved the problem by changing the way the application I was running in the Container worked. Now it uses no swap.

You can find out what is using swap in a number of ways. There are some suggestions here: https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/ (scroll down to Listing all processes using swap space)

About 64GB free ram at the host-system.
I can't change anything in the containers, because those are customers containers.

Best regards

 

[TechLineX]

root@host:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           125G         69G        1.3G         13G         55G         41G
Swap:          8.0G        8.0G          0B

Still the same issue..

 

[Faris Raouf]

I'm not sure how to test things in any significant detail, but I have discovered that there are three things that seem to affect how much swap gets used in my situation (which is different to yours but maybe there is a link)

1) I/O wait
2) Network traffic
3) Amount of free RAM on the Node.

Regarding item 3, I have two Nodes that have identical hardware and configuration. One has 50% free RAM with RAM icon green, one has 40% free ram with the RAM usage icon in the GUI is orange/amber.

When performing an rsync between the nodes that causes up to 10% I/O Wait and up to 100% Ethernet bandwidth usage on the node with 40% free RAM, swap increases significantly, and would continue to increase to 100% if the task I'm running ran long enough.

On the node with 50% free RAM, no additional swap is used.

And this is why I think items 1,2 and 3 are linked to how much swap is being used.

If the tasks in your Containers are causing high disk I/O wait, or high network traffic, or maybe if you have a complex firewall set up in the Node, it is possible one of these things may be triggering the swap usage.

What I suggest you try is to reduce swappiness on the Node, maybe to 10. If you do a search of these forums, you'll find several threads discussing this issue.

The basic setting can be found here: https://pve.proxmox.com/wiki/ZFS_on_Linux (scroll down to swappiness. It doesn't matter that this page is talking about ZFS - ignore that).

But it seems that there is swappiness and there is cgroups swappiness. Take a look at https://forum.proxmox.com/threads/swappiness-question.42295/

The importance of the cgroups swappiness is that it may be that something is causing the proxmox firewall or some other proxmox element that uses cgroups swapiness settings to use swap, rather than the swap being caused directly by whatever is running in the Container itself.

** Please be aware that I only have the most basic understanding of what is happening here. All I can do is try to provide more information that may help you. I am still seeking answers myself, mainly by experimentation **

 

[TechLineX]

Is there someone at the proxmox-team who made the same experience??

Swappiness was at 10 in the past, but I determined various random crashes of my hosts. So I set it back to the default value. What I can confirm is, that, if I copy large files, f.ex. when migrating the storage of a vps, the swap increases very strong. After the migration it falls back.

The problem is, I have about 100-150 lxc-containers and I can't monitor their I/O-Load. Is there an easy tool or workaround to do this??

Best regards

 

[TechLineX]

#

 

[mailinglists]

I did not read whole thread, but here is an idea.
Since SWAPping on ZFS makes your system unstable, either disable SWAP completely or move it off ZFS.

 

[TechLineX]

I did not read whole thread, but here is an idea.
Since SWAPping on ZFS makes your system unstable, either disable SWAP completely or move it off ZFS.

Problem is, that if I disable the Swap partition, the system randomly reboots..

 

[TechLineX]

#

 

[TechLineX]

Isn't there anyone on the proxmox team here who wants to take this case? It´s still the same. When I use LXC-Container, my swap is running out of space and the whole system slows massivly down.

 

[TechLineX]

#

 

[TechLineX]

#

 

[Faris Raouf]

I think everyone has run out of ideas.

Have you looked at https://forum.proxmox.com/threads/swappiness-question.42295/ recently?

The key issue with your problem is that you are experiencing slowdowns or worse when your swap fills. This should not happen UNLESS you are also running out of physical RAM somewhere, somehow (although it is clear you have lots of RAM). I still think that might be the key thing here -- find out what's using your swap. The topic above has some ideas on how to do that, along with some of the other links I posted earlier. My gut feeling is that this may be key to solving your problem.

 

[TechLineX]

Would be nice, if the proxmox team would give a comment on that.

Best regards

 

[mbaldini]

Have you tried to move swap to RAM with ZRAM?

 

[mailinglists]

If you disable SWAP and then your system becomes unstable, it might be, because you run out of RAM (memory) and it would also make sense that it becomes slow when SWAP is enabled, because it uses more memory than you have RAM, thus begins SWAPing.

When looking into RAM usage, also take a look at or maybe limit the amount of buffered memory used for file cache and look for colleration of amount of buffered memory vs system speed.

I would monitor memory usage with "free", find the memory hog or memory leak, try to fix it if possible and maybe upgrade the RAM, if there is no memory leak, and you actually need more RAM, so it does not SWAP.

Also I do not see how, setting to system to swap later (swappiness) can make the system unstable. I would recheck how you came to this assumption.

Good luck!