[SOLVED] SWAP 100% load

[lDemoNl]

Hi guys! Few days ago I've noticed that SWAP on 1 of 5 hosts was loaded for 100%. On this host I used command swapoff -a && swapon -a and everything was OK. But today situation repeated. How to solve this situation? I can't every 3-4 days disable than again enable swap.
I use pve-manager/6.2-11/22fb4983

 

[apoc]

If a system swaps then I would say it is running into an 'out of memory" condition. The question for me would be: what is the source?
Are the machines under high memory pressure?
What value do you use for swappiness?

 

[lDemoNl]

If a system swaps then I would say it is running into an 'out of memory" condition. The question for me would be: what is the source?
Are the machines under high memory pressure?
What value do you use for swappiness?

What do you mean by source and memory pressure?
This is my system on picture. 4 another hosts the same (They use 50MB-1500MB of swap). On this host ~13 running VM

 

[apoc]

OK. Looks like you have plenty of free memory.
So the question is: why does your system swap? I can see two easy reasons:
Option 1: ZFS ARC - do you use ZFS?
Option 2: VM.swapiness value is high (default is 65 IIRC) - if you have not changed it (not sure if Proxmox does this on install) your system might start to swap even if there is enough memory.
Please check this: https://askubuntu.com/questions/103915/how-do-i-configure-swappiness
I have set mine to 1 (%)

 

[lDemoNl]

OK. Looks like you have plenty of free memory.
So the question is: why does your system swap? I can see two easy reasons:
Option 1: ZFS ARC - do you use ZFS?
Option 2: VM.swapiness value is high (default is 65 IIRC) - if you have not changed it (not sure if Proxmox does this on install) your system might start to swap even if there is enough memory.
Please check this: https://askubuntu.com/questions/103915/how-do-i-configure-swappiness
I have set mine to 1 (%)

1) I don't use ZFS
2) I didn't change swapiness (default 60). Am I right, that I have VM with a lot of free RAM? So Host moved this free RAM to SWAP?

 

[apoc]

If it is not ZFS and you still have a high swapiness value then this is it.
You ar e way above 60% free memory so it will use your swap.
Set it to 1, upgrade initramfs and reboot. I guess your behaviour will go away

 

[lingfish]

If it is not ZFS and you still have a high swapiness value then this is it.
You ar e way above 60% free memory so it will use your swap.
Set it to 1, upgrade initramfs and reboot. I guess your behaviour will go away

No no no, it's not that simple.

Please, please read here: https://www.howtogeek.com/449691/what-is-swapiness-on-linux-and-how-to-change-it/

 

[apoc]

Thanks mate. No wonder everyon tells that wrong wisdom. The whole explanation is looooong.
Will read it though and educate myself.
Damn. It sounded so obvious...

 

[charleslcso]

I have this now... would PVE crash if swap is full? Any solution?

 

[Kingneutron]

You still have plenty of free RAM. I solved this for my homelab by running a script every hour that does swapoff -a; swapon -a

 

[waltar]

Set swappiness to 0, how to example here https://gist.github.com/docapotamus/f7906b529c940e48bd2b

 

[alexskysilk]

I have this now... would PVE crash if swap is full? Any solution?

It wont. swap is there to be used.

 

[waltar]

PVE on zfs don't have swap configured at all (as it's a problem of swap on zfs) but there it's even better to have as on non-zfs pve because arc cannot release as fast cache as normal linux fs cache do and so an oom is more likely ...
But I don't like swap, so disable in fstab otherwise I could use any more 10years older hosts than today also when have the time to wait on swapping ...

 

[charleslcso]

It wont. swap is there to be used.

Does anyone know what happened during the "swap off" process? CPU and IO are blank. I checked the syslog during that period of time, and may be I missed some key info, nothing weird during that time.

Oh, I'm not using ZFS.

 

[charleslcso]

Set swappiness to 0, how to example here https://gist.github.com/docapotamus/f7906b529c940e48bd2b

Thank you for this info. I've just added this line to the PVE host.

Wait a tic. I really don't understand this.

Is this swap the sum of all swaps in VMs and LXCs? Should I update every VM and LXC to vm.swappiness=0?

PS: just checked the 1 VM and all LXCs, most of them are either 0 swap or less than 3%. So this answered my own question.

 

[waltar]

Set swappiness just on pve host as vm/lxc are much smaller in mem and so maybe need it which should better not be the case of the host.

 

[alexskysilk]

Does anyone know what happened during the "swap off" process?

swap gets drained back to ram and disabled.