Clear cache buildup inside VMs

harmonyp

Member
Nov 26, 2020
195
4
23
46
Hello,

To optimize in getting more out of the limited RAM my server has I plan on doing the following to clear cache build up:

Linux hourly cronjob to run
root sync; echo 3 > /proc/sys/vm/drop_caches

Windows hourly schedule to run EmptyStandbyList.exe (https://wj32.org/wp/software/empty-standby-list/)

I presume there are downsides to doing this but while doing this myself manually I can't even detect any performance drop, is it negligible? Also do you guys suggest doing anything else other than KSM + Ballooning
 
Last edited:
If you can get a lot of memory back from dropping caches inside the VM, why not just give the VM less memory? And maybe add more swap inside the VM also.
 
If you can get a lot of memory back from dropping caches inside the VM, why not just give the VM less memory? And maybe add more swap inside the VM also.
Not an option as they might need the extra RAM at peak times. Problem I see is the RAM buff/cache doesn't seem to ever clear. They don't have swap I will see how that helps.
 
Not an option as they might need the extra RAM at peak times. Problem I see is the RAM buff/cache doesn't seem to ever clear. They don't have swap I will see how that helps.
You make sure that peaks don't happen for multiple VMs at the same time? Linux (or Windows) has no reason to drop caches (or read back swap to memory) unless it needs it for something. As far as it knows, it might be usefull in the future. Write cache can be disabled by using sync as a mount option in /etc/fstab. I would expect that when the balloon inflates to give memory back to Proxmox (and other VMs), Linux would just drop the (read) caches for you. Maybe the ballooning of other VMs does not happen fast enough to prevent another VM from running out of memory? Maybe someone knows a way to make ballooning happen more often to distribute the memory more quickly to VMs that need it?

EDIT: Using something like pvesh create "/nodes/${HOSTNAME}/qemu/${DOMAIN}/monitor" --command "balloon ${SIZE}" you can influence the size of the balloon inside VMs from the Proxmox host. Replace ${DOMAIN} with the VM-number and ${SIZE} with anywhere between the minumum amount of memory and the maximum amount of memory to determine the amount of memory available to the VM.
 
Last edited:
I'm still waiting for some kind of intelligent balooning. Lets say I got this scenario:
Host: 32GB RAM (4GB used by host itself)
Guest1: 8GB RAM (max), 4GB RAM (min) -> 20% used, 80% cache, 0% free
Guest2: 8GB RAM (max), 4GB RAM (min) -> 80% used, 20% cache, 0% free
Guest3: 8GB RAM (max), 4GB RAM (min) -> 50% used, 0% cache, 50% free

Example above is RAM used 4G (host) + 8GB (Guest1) + 8GB (Guest2) + 4GB (Guest3) = 24GB RAM or 75% of hosts RAM used.
As long as hosts RAM will not exceed 80% balooning will not kick in. And each VM may use full 8GB RAM.

Now lets say the host increases its RAM usage by 2GB from 4GB to 6GB. Hosts RAM usage would now be 26 GB or 81.25% and balooning kicks in.
If all VMs got the same balooning ratio, balooning will limit guests1 RAM to 4GB, guests2 RAM to 4GB, and guests3 RAM to 4GB.
For guest3 that is no problem because half of the RAM is free.
For guest1 that is also no problem because 80% of RAM is used for caching and the VM drops the caches in no time to that point where the guests only got 20% used, 30% cache, 50% free.
But a big problem is guest 2. Here blooning limits the RAM from 8GB to 4GB but 6,4GB are used by processes so many of the processes much be killed by OOM so that everything fitsin 4GB again.

That is very stupid that processes of guest2 get killed by ballooning while guest1 is wasting memory at the some time by still using caching.
As long as ballooning isn't intelligent enough to dynamically reduce the RAM by useage (for example only -10% for guest2 but -70% for guest1 and -40% for guest3) instead of fixed ratios overcommiting RAM just isn't working and you shouldn't assing more RAM to VMs than your host got available.

Only option I see is to buy more RAM or limit the RAM of each VM to that point that it is just enough to work most of the time. But then there is no headroom and every peak hits the limit and causes alot of swapping.
I really miss an option in linux to limit the cache size like I can do it with ZFSs ARCs "zfs_arc_max" option. Completely dropping the cache is bad because it will increase performance. But unlimited cache will waste a lot of RAM other VMs could do better use of. Would be so great if I just could limit the page file caching to 1 or 2 GB.
 
Last edited:
This is a mooe point. Some workloads cannot be run as containers. Proxmox Backup Server is one of the examples and it is a heavy hitter when it comes to fs caches. I have it running on my cluster and it consumes 1GB for itself and 5GB for cache. The performace gaude in proxmox is red and if I wouldn't know better I would be worried... I agree with @Dunuin that the balooning driver could do a better job considering that it has good picture of memory usage and should be able to clearly differentiate memory use by applications vs. caches and buffres.
 
This is a mooe point. Some workloads cannot be run as containers. Proxmox Backup Server is one of the examples and it is a heavy hitter when it comes to fs caches. I have it running on my cluster and it consumes 1GB for itself and 5GB for cache. The performace gaude in proxmox is red and if I wouldn't know better I would be worried... I agree with @Dunuin that the balooning driver could do a better job considering that it has good picture of memory usage and should be able to clearly differentiate memory use by applications vs. caches and buffres.
Thats the problem. How the qemu guest agent is implemented now it won't report to the host how the guest is using that RAM, so PVE can't know if a VM is using most cache for buffering or for system/userspace processes. Another problem is that the KVM process won't give any RAM back to the host if the guests frees up the page file cache. If I got a VM with 8GB RAM it for example starts with 2GB used + 6GB free and the KVM process uses 2GB RAM on the host. Then I do some IO and the cache will climb up until its 2GB used + 6GB cached and the KVM process on the host will increase to 8GB. But if the guest then frees up 6GB of cache (so it is 2GB used + 6GB free again) the KVM process will still consume the full 8GB RAM until I shutdown that VM. Not sure if it will be better with ballooning enabled, but without KVM will waste alot of unused memory that way.
 
Last edited:
[...] Not sure if it will be better with ballooning enabled, but without KVM will waste alot of unused memory that way.
I would expect that with ballooning Proxmox will get to use that 6GB for other things. I don't know if the kvm process will actually release it or whether it will show as shared or something.
Note that ballooning will only give memory to VMs up to 80% host memory usages (which includes the ZFS ARC, unfortunately). Once your host uses over 80%, your VM cannot use that 6GB anymore (gradually) even when there is memory free on the host.
 
I mean something like this:
ram03.png

Hosts KVM process (1.6G RES) and free -h inside VM (12G free) when starting VM:
ram01.png
ram02.png

Doing some stuff in VM increasing RAM usage. KVMs goes up to 10G RES, free RAM in VM goes down to 6.5G:ram04.png
ram05.png

Stopping stuff in VM. Free RAM in VM goes up to 10GB again, the KVM process will stay at 13.1G or might even get higher but will never get smaller unless I restart that VM. Thats the same for all my 30 VMs running Linux/Win/Unix which all got ballooning drivers and qemu guest agent installed. Also no matter if something is PCI passthroughed or not.
ram06.pngram07.png

In that case above I also can't set a the VMs "Minimum Memory" to below 16GB because that VM is running 5 Minecraft-Servers and if I start all 5 of them at the same time the VM really needs the full 16GB or the guest will OOM. But sometimes I just want to run 1, 2 or 3 Minecraft-Servers at a time and the VM is wasting 10 GB of RAM because the KVM process will stay somewhere at 14-16GB even if 10-12 GB are actually "free" and at the moment not used by the guest.
 
Ah right, I misunderstood. Maybe hot-plugging memory might help. I believe memory unplugging also works on modern Linux VMs. Not sure how to automate this but it should allow you to change the memory according to the needs to the VM.
Alternatively, run each Minecraft server in a separate VM (enable KSM to share some memory maybe?) and start and stop entire VMs according to your needs?
 
  • Like
Reactions: Dunuin
I already use KSM which is saving me around 9GB of my 64GB RAM. Running all Minecraft server as individual VMs would be annoying because I like to use MineOS as a WebUI to manage and monitor those Minecraft servers and in that case I wouldn't be able to manage all of them through a single WebUI. And all the 5 Minecraft server are connected as a bungee network with a shared MySQL DB as a backend for all the Minecraft mods. So then I also would need to put the MySQL and the bungee proxy that connects all server together in another VM. So because everything is connected and works as a whole I decided to use a single VM for it which works really fine for over a year except for that RAM usage that can only go up but never down again.

Will read a bit about memory hot-plugging. Didn't used that yet.
 
I already use KSM which is saving me around 9GB of my 64GB RAM. Running all Minecraft server as individual VMs would be annoying because I like to use MineOS as a WebUI to manage and monitor those Minecraft servers and in that case I wouldn't be able to manage all of them through a single WebUI. And all the 5 Minecraft server are connected as a bungee network with a shared MySQL DB as a backend for all the Minecraft mods. So then I also would need to put the MySQL and the bungee proxy that connects all server together in another VM. So because everything is connected and works as a whole I decided to use a single VM for it which works really fine for over a year except for that RAM usage that can only go up but never down again.

Will read a bit about memory hot-plugging. Didn't used that yet.
How did the reading/test go with hot-plugging memory?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!