VM swap best practice

[MMartinez]

Hi,

I'd like to ask about the best practice on KVM guest configuration to avoid host swap usage and have the best performance.

On virtualitzation systems I liked to remove the swap partition on my VMs and give enought resources to the VMs and Hosts, but this can be sometimes problematic, so I use this strategy just sometimes, with concrete VM that I don't want to page memory on slow disk (and I monitor them to be sure that the assigned memory is enought).

In some of my proxmox nodes, despite having enought memory, I see a high usage of swap (> 90%). For example there is one with 128GB and 48% of RAM usage which is using 90.38% of swap (7.23GB over 8GB available).

Using a command I found time ago in https://stackoverflow.com/questions/30481314/how-can-i-know-which-process-is-using-swap, I can see which are the VMs that use more swap, so the first thing I usually do to balance the swap usage on the hosts is to move some of the high swap demanding VMs to another node (I've seen also that reducing the swappiness on the node to 10 makes the live migration process faster in the case that the VM is using a lot of swap, because the host is more agressive recovering swap, but this is another point). The command I use is:

(echo "COMM PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r)

I have been searching information about the best practice on swap usage on the VM and one post, by Christopher Pereira on https://unix.stackexchange.com/questions/137509/allocating-swap-space-with-kvm, seems interesesting for me. It's a different aproach:

"You should also enable swap space on the VM, so it is aware that RAM is being swapped and avoid using it for disk cache and IO buffers.

If you only enable swap on the host, the VM will believe it has available real RAM and will use it, while it will be swapped on the host creating unnecessary bottlenecks and freezes. This is the biggest mistake you could do.

You should also use a different virtual drive as a swap disk, since you don't want to snapshot/backup it together with your other data drives.

Since swap space can be volatile, you could also use cache=unsafe or use zram/zswap for better performance.

Also remember that swapping is good, since it frees inactive RAM that can later be reused for disk cache and IO buffers."

What do you thing about this subject? Is Christopher Pereira right?

Regards,

Manuel Martínez

 

[leesteken]

"You should also enable swap space on the VM, so it is aware that RAM is being swapped and avoid using it for disk cache and IO buffers.

Indeed. Only the operating system inside knows what the memory is use for and can swap out rarely used memory for I/O buffers, which is a good thing (and can be tuned).

If you only enable swap on the host, the VM will believe it has available real RAM and will use it, while it will be swapped on the host creating unnecessary bottlenecks and freezes. This is the biggest mistake you could do.

Yes, swapping out a VM on the host is a performance disaster. People always advise to not overcommit memory when virtualizing.

You should also use a different virtual drive as a swap disk, since you don't want to snapshot/backup it together with your other data drives.

Good idea and also for /var/tmp/ (and maybe others?); both don't need to be backed up.

Since swap space can be volatile, you could also use cache=unsafe or use zram/zswap for better performance.

Yes, but this might cause it to stay in host memory for a longer time, which causes extra memory pressure on the host, which could be better used for VMs. ZFS swap is usually set to sync=always to move it to disk as soon as possible instead of buffering it and defeating the purpose.

Also remember that swapping is good, since it frees inactive RAM that can later be reused for disk cache and IO buffers."
What do you thing about this subject?

Writing to swap is a good thing (except maybe for SSD wear). Reading from swap is a bad thing and indicates that you need more memory. As long as you don't have more memory than you need, some swap is probably always a good idea.

 

[MMartinez]

Indeed. Only the operating system inside knows what the memory is use for and can swap out rarely used memory for I/O buffers, which is a good thing (and can be tuned).

Yes, but this might cause it to stay in host memory for a longer time, which causes extra memory pressure on the host, which could be better used for VMs. ZFS swap is usually set to sync=always to move it to disk as soon as possible instead of buffering it and defeating the purpose.

Using a dedicated swap disk on the VM would be a good way also to reduce the size and time of incremental backups using PBS, so it seems a win win choice.

Setting the VM disk options to cache=unsafe can cause to be swapped on the host? I wasn't aware of that. What other things can cause the use of swap in the host?

Regards.

Manuel Martínez

 

[leesteken]

Setting the VM disk options to cache=unsafe can cause to be swapped on the host? I wasn't aware of that. What other things can cause the use of swap in the host?

Enabling write-caching means data is kept in memory longer and written later (unsafe also does this for sync writes and lies about them being written to disk immediately). Therefore more memory is in use, which might cause swapping. Swap usage can be caused by anything that allocates memory really. Or when doing I/O which might be buffered or cached (when Linux decides that memory is better used for cache than applications).

 

[MMartinez]

Enabling write-caching means data is kept in memory longer and written later (unsafe also does this for sync writes and lies about them being written to disk immediately). Therefore more memory is in use, which might cause swapping. Swap usage can be caused by anything that allocates memory really. Or when doing I/O which might be buffered or cached (when Linux decides that memory is better used for cache than applications).

Yes, I knew that, but I thought that syncs to disk should be as quick as possible and therefore it would not be needed to use swap for that.

 

[kripper]

Enabling write-caching means data is kept in memory longer and written later (unsafe also does this for sync writes and lies about them being written to disk immediately). Therefore more memory is in use, which might cause swapping.

Yes, but IMO this also means that the swapped memory on the VM will actually not be written (immediately/ever) to physical disk on the host but kept in RAM.

This way, the decision to actually swap to physical disk (which may cause hiccups on the host and on all VMs) will be handled entirely by the host, which knows that this data is only a cache and can be freed anytime.
What is more, in case you "undercommit" memory and the VM swaps memory, the host would still keep this data in RAM and only swap to physical disk when really needed (when the overall RAM on the host gets low).

It would be interesting to test a VM with very low memory and measure the performance on the host when the VM swaps memory. If everything is kept cached in RAM on the host until more memory is required, I would assume this is the optimal way to handle swapping. Otherwise, if the VM swap disk cached data is not released on the host when RAM gets low, I would say something is not correctly implemented in QEMU-KVM.

BTW, this is a critical undocumented and unclear topic that we should maybe discuss directly on StackExchange so other people can comment and collaborate without having to register on this forum (as I did).