PVE uses swap while RAM is*enough, why?

[achekalin]

Recently I got a chance to install PVE 3.3 on quite a capable server (Dual 6-core Xeons, 128 RAM, SSD disks). Due to fact it has a lot of RAM I set up swap to only 8Gb thinking I'll newer need any (and I hate to spend SSDs for swap, too). I created and run some VMs, and as I finished I have 126 out of 128 GB assigned to machines. Guest OSes are Ubuntu server 12.04 and Windows 2008 R2.

To my great surprise, after only two days of running I see that swap usage on hast server is 6 Gb, and I see no reason why it is so.

The free -m command says this:

             total       used       free     shared    buffers     cached
Mem:        128902     124397       4504          0        244        142
-/+ buffers/cache:     124010       4892
Swap:         8191       1334       6857

So I see 6.8 Gb of swap out of 8 is used now, quite a amount!

I can play with swappiness variable, but should I do that or I'd better decrease RAM assignment on some of VMs to have some spare free RAM? Then how much RAM should I preserve and keep 'just in case'?

I found no recommendation on that subject, but I think the question is serious: there are a lot of PVE servers nowdays uses SSDs (or something SSD-backed) as a storage, so wasting it as swap will wear it out and anyway decreases system speed.

Thank you in advance for any help on this matter, PVE is great system and I hope this turns out to be easily resolvable question!

 

[dietmar]

So I see 6.8 Gb of swap out of 8 is used now, quite a amount!

No, only 1.3GB are swapped.

 

[achekalin]

No, only 1.3GB are swapped.

You're right. I saw 6 Gb some time ago, so it appears to fluctuate.

But anyway: why even 1 Gb of swap used while I have much more RAM as 'free'? Any approaches to free swap completely? Lower overall RAM assignment, zRAM?

 

[mo_]

I'm afraid this is a major design flaw in both linux and windows alike: if you give it swap space, it'll use it, regardless of the amount of free ram available. For linux this even tends to happen if you mess around with the swappiness tunable.

 

[achekalin]

I'm afraid this is a major design flaw

Sounds quite correct!

Just out of curiosity: may this be a good idea to implement swap made of zRAM block device(es)? It's common practice when we come to notebook with large RAM (which is not uncommon today) and SSD disk - to prevent SSD wear out and seepd things up a bit it's a good idea to use zRAM-based swap.

 

[mijanek]

I'm afraid this is a major design flaw in both linux and windows alike: if you give it swap space, it'll use it, regardless of the amount of free ram available. For linux this even tends to happen if you mess around with the swappiness tunable.

no I wouldn't say it that way...:

root@proxmox:~# free -m
             total       used       free     shared    buffers     cached
Mem:         16032      15869        163          0        816       5780
-/+ buffers/cache:       9272       6760
Swap:        15231          0      15231

root@proxmox:~# uptime
 18:38:09 up 8 days, 20:20,  1 user,  load average: 0.20, 0.23, 0.26

This is likely, that you may temporarily used more RAM than it was avaiable, therefore part of it has been swapped out, and as long there is no need of it, it will remain swapped.

 

[achekalin]

Yes it looks like that. But given the fact I use th same number of VMs, and all of them are configured once and never touched, and sum of all RAM assigned is less that total RAM amount that's installed in host machine, and I finally got 'out of RAM' which given me swap usage, then I really wanted to know how mocu RAM should I keep in mind to be reserved when configure VMs.

Say 128 GB is plenty, so how much should I keep untouched: 4, 6, 8 GB (thus limit total for VMs to 124, 122, 120 Gb), or maybe even more? Any idea how much should this number be, and if it differs for host with 64 or 128 or 256 Gb of RAM?

 

[achekalin]

Funny thing: I tried to set up 4 zRAM block devices each of 1,5 Gb, and added it as swap with high priority. I set vm.swappiness = 10. After some hours with this config, I see these numbers:

# free -m
             total       used       free     shared    buffers     cached
Mem:        128902     125280       3621          0        295        147
-/+ buffers/cache:     124837       4064
Swap:        14335       5681       8654

and

# swapon -s
Filename                                Type            Size    Used    Priority
/dev/mapper/pve-swap                    partition       8388600 954672  -1
/dev/zram0                              partition       1572856 1215592 10
/dev/zram1                              partition       1572856 1215976 10
/dev/zram2                              partition       1572856 1215764 10
/dev/zram3                              partition       1572856 1216124 10

So I just can't understang, I see 3Gb of free ram, and I use 1,5*4 Gb RAM as zRAM devices - that amount is quite a lot to hold all that swap space used in my original post. But anyway I see swap is used. What I missed?

 

[macday]

Please don´t use zram on proxmox. This will slowdown the whole system on swap usage.

Just put a vm.swappiness = 0 in your /etc/sysctl.conf and reboot the host.

 

[achekalin]

Please don´t use zram on proxmox. This will slowdown the whole system on swap usage.

But if I have a lot of free memory, won't it be wise to keep some spare swap space made of zram (thus fast one, not like ordinary hard drive, even in raid)? I understand that if I have free memory then I should not need swap at all but as I used host 128 GB of RAM and assign 126 Gb to VMs I got some used GB of swap, so now I'm in doubt how much ram keep free so no swap be used at all.

Just put a vm.swappiness = 0 in your /etc/sysctl.conf and reboot the host.

Will try, thanks!

 

[gerdnl]

that wont work with a privilaged LXC container

 

[jason0]

I have this problem too. I set swappiness to 0 and it will still occasionally try to swap even though I have plenty of free RAM.
And anytime it tries to swap, it brings my system to a halt. Can't do anything!
VM's that normally take a few seconds to boot, takes minutes instead.

I feel like there is a different underlying problem, but performing 'swapoff -a' is a good solution for me.

 

[gerdnl]

if u put swapiness to 0 ur disabling the swap and ur system will crash when its out of memory , put it on 1 or 10

 

[gerdnl]

recently swapiness 0 has become to disable swap and 1 is now the same as (only swap when needed)

 

[gerdnl]

anyone from proxmox able to clear this up?

 

[gerdnl]

no answer from proxmox ?

 

[Petr]

Occasionally have this issue. Happened before on PVE 5.0 and happens now on recent PVE version.
Intensive swap IO correlates to backup (vzdump) process. Processes, that don't release swap memory, are kvm running Windows.

# free -m
total used free shared buff/cache available
Mem: 290204 158568 1493 1113 130143 137079
Swap: 8191 8033 158

# pveversion
pve-manager/5.1-46/ae8241d4 (running kernel: 4.13.13-6-pve)

 

[Rhinox]

I really do not understand this whining about swap over and over again...
If you do not want to see swap being used, simply buy more ram and disable swap completely!