Adding more swap without reboot. Running into some issues

[daniel_wwf]

Hey there,

I have several hosts running with around 70 - 100 LXC's each. On 2 of the Hosts, I am seeing the swap of 6/8 GB being used almost 100%. I reduced swappiness to 0 on two of the systems and used swapoff -a to kill the swap. This fails, The system is reducing the swap to a certain point where its no longer able to do so and ill end up with a swap usage of like 1GB out of 6GB. Which was fine for me. After 24h I figured this is causing stalling daemons on some of the LXC's. So having no swap on the host is no solution for me.

I tried to increase the swap the usual way I am used to it from "normal" distros, but this fails as well. I have noticed the existing swapfile is symlinked for at least two locations. I am now unsure how to do this correctly. Is there any sort of guide to follow or am I missing something? I can NOT reboot the systems and having no swap is NOT a solution for me. Any downsides like loss of performance won't matter as I wouldn't notice them anyways as long as everything is running stable.

The Hosts have 128GB RAM where around 50GB are in use. So it's not an issue of running out of RAM

This is what i usually do to add more swap:

1. Create the file. This command creates a 1 gigabyte file: dd if=/dev/zero of=/swap bs=1M count=1000
2. Format the swap file: mkswap /swap
3. Add the new swap file to /etc/fstab: /swap swap swap defaults 0 0
4. Run swapon -a to activate the new swap.

Swapon fails with: swapon failed: Invalid argument.

Googled it and tried a lot of suggestions, still not working.

Any help appreciated!

 

[wolfgang]

Hi,

what you get an error if you call it directly with this call?

swapon -v </path/to/swapfile>

The Hosts have 128GB RAM where around 50GB are in use. So it's not an issue of running out of RAM

Do you use ZFS or any kind of DB in the container?
Because they have huge metadata memory sections what will not be used.
After a while, the Linux kernel decided to swap it.
So this is no matter of having no Memory left.
It is more a do you don't use this memory section so the kernel swaps it.

 

[daniel_wwf]

Hey Wolfgang,

thanks for your reply. If i do this on a test system, this is what happens:

root@xxx /dev/pve # swapon -v /dev/pve/swap1
swapon: /dev/pve/swap1: found signature [pagesize=4096, signature=swap]
swapon: /dev/pve/swap1: pagesize=4096, swapsize=6442450944, devsize=6442450944
swapon /dev/pve/swap1
swapon: /dev/pve/swap1: swapon failed: Invalid argument

We don't use ZFS and the LXC's are running some sort of Berkeley DB. We did some tests and figured, it's not an issue of having the system using swap. It's more that we didn't create swap big enough. The performance of individuals LCX's turns out to be worse if we avoid swapping at all.

 

[daniel_wwf]

Okay... if i do:

fallocate -l 1G /swap4
chmod 600 /swap4
mkswap /swap4
swapon -v /swap4

It works! It seems I can't do this in /dev/pve. This means I am putting the file in the wrong place.

I did extend the LV

 --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                zMA6Wp-sekg-PO4W-u88H-fBdz-jpov-tPdmcf
  LV Write Access        read/write
  LV Creation host, time rescue, 2018-11-05 05:26:02 +0100
  LV Status              available
  # open                 2
  LV Size                30.00 GiB
  Current LE             7680
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

Was this wrong? My idea was: I am going to increase the LV and will put the file there. I then got confused because of all the symlinks and was like, okay lets do it in ./pve then. So the real question seems to be: where to put it?