Why set minimum RAM when you have Ballooning + KSM enabled?

[harmonyp]

I am a little confused by what the purpose of setting minimum RAM is when you have ballooning + KSM enabled.

If a virtual machine has 1024/2048 it can always use 2GB of that and even if it was using 512MB it would show the exact same usage either way (on the VM + Overall node usage)

I've been told it allows overselling but that's exactly what ballooning + KSM does so I am not sure if I should set minimum RAM. Can someone please explain the advantages/disadvantages.

One disadvantage I have come across when setting the memory low is Linux machines don't show the correct memory (top) and sometimes show no memory errors so I am thinking about removing it.

 

[Dunuin]

As far as I understand ballooning only works if your RAM of the host it over 80% used and it will only limit guests RAM between min and max if you set a min value. KSM will also start at 80% RAM usage by default config but you can change that.

 

[harmonyp]

As far as I understand ballooning only works if your RAM of the host it over 80% used and it will only limit guests RAM between min and max if you set a min value. KSM will also start at 80% RAM usage by default config but you can change that.

So setting the minimum low only lets you not use KSM faster? What's the advantage of that?

But Ballooning doesn't go towards the 80%? If the total nodes RAM is 32GB and you give 10 virtual machines 1024/4096 OR 4096/4096 but none of them are actually using more than 1GB then the total usage will still be 10GB (1/3 of total RAM)???

 

[Dunuin]

So setting the minimum low only lets you not use KSM faster? What's the advantage of that?

But Ballooning doesn't go towards the 80%? If the total nodes RAM is 32GB and you give 10 virtual machines 1024/4096 OR 4096/4096 but none of them are actually using more than 1GB then the total usage will still be 10GB (1/3 of total RAM)???

No, I mean balooning changes the available maximum usable RAM of the guests. If you set a VM to min=1GB and max=4GB ballooning will do nothing until your hosts RAM is 80% full. After that it will limit the maximum usable RAM of the guest from 4GB down to 1GB. If you set min=2GB the guests maximum usable RAM will change between 2GB and 4GB. If you don't set a lower min RAM value it shouldn't balloon at all.

KSM is another thing that works independently but is also preconfigured not to work until your hosts RAM exceeds 80% usage.

If the total nodes RAM is 32GB and you give 10 virtual machines 1024/4096 OR 4096/4096 but none of them are actually using more than 1GB then the total usage will still be 10GB (1/3 of total RAM)???

Until more then 25,6GB (80%) of you hosts RAM is used all 10 VMs will have a maximum 4GB RAM in the guest OS. If your hosts RAM exceeds 25,6GB the ballooning starts working and will limit your guests maximum RAM. If set to 1024/4096 and the ballooning ratio is everywhere the same, then it should set the max RAM of all of your guests to 3,2GB. Id you set it to 4096/4096 nothing should happen because no guest is allowed to have less that 4GB RAM.

 

[harmonyp]

No, I mean balooning changes the available maximum usable RAM of the guests. If you set a VM to min=1GB and max=4GB ballooning will do nothing until your hosts RAM is 80% full. After that it will limit the maximum usable RAM of the guest from 4GB down to 1GB. If you set min=2GB the guests maximum usable RAM will change between 2GB and 4GB. If you don't set a lower min RAM value it shouldn't balloon at all.

KSM is another thing that works independently but is also preconfigured not to work until your hosts RAM exceeds 80% usage.


Until more then 25,6GB (80%) of you hosts RAM is used all 10 VMs will have a maximum 4GB RAM in the guest OS. If your hosts RAM exceeds 25,6GB the ballooning starts working and will limit your guests maximum RAM. If set to 1024/4096 and the ballooning ratio is everywhere the same, then it should set the max RAM of all of your guests to 3,2GB. Id you set it to 4096/4096 nothing should happen because no guest is allowed to have less that 4GB RAM.

Thanks for the reply I get the first part now.

I am confused by the part where you said no guest will have less than 4GB of RAM if set to 4096/4096. If that's true then what's the point of enabling ballooning at all? I have read up on ballooning and from what I understand rather than if a VM uses up the whole 4GB and remains cached ballooning will basically clear the cache and give the RAM back to the host once finished?

 

[Dunuin]

Memory ballooning (KVM only) allows you to have your guest dynamically change it’s memory usage by evicting unused memory during run time. It reduces the impact your guest can have on memory usage of your host by giving up unused memory back to the host.

The Proxmox VE host can loan ballooned memory to a busy VM. The VM decides which processes or cache pages to swap out to free up memory for the balloon. The VM (Windows or Linux) knows best which memory regions it can give up without impacting performance of the VM.

Ballooning frees up cached stuff in the guests RAM because it forces it by lowering the RAM available to that VM. If the VM got 4GB RAM available and processes use 1GB of RAM the VM will use the remaining 3GB of RAM for caching. If ballooning kicks in and lower the available RAM of the VM from 4 to 2 GB, the VMs needs to free up half of its RAM so it can be removed from that VM. So ballooning basically changes the size of the available RAM.

 

[NanamiMio]

Ballooning frees up cached stuff in the guests RAM because it forces it by lowering the RAM available to that VM. If the VM got 4GB RAM available and processes use 1GB of RAM the VM will use the remaining 3GB of RAM for caching. If ballooning kicks in and lower the available RAM of the VM from 4 to 2 GB, the VMs needs to free up half of its RAM so it can be removed from that VM. So ballooning basically changes the size of the available RAM.

Thank you for your reply.
If I understand correctly, the free up process does not happen when the RAM is set to 4096/4096. So 4096/4096 is equivalent to disabling the ballooning?

 

[Dunuin]

Thank you for your reply.
If I understand correctly, the free up process does not happen when the RAM is set to 4096/4096. So 4096/4096 is equivalent to disabling the ballooning?

Yes.

 

[corndelphinus]

Hello, i have 3 VM´s (Windows 11 with all the drivers https://pve.proxmox.com/wiki/Windows_10_guest_best_practices)

My latests Proxmox 8.2.2 node have 32 GB RAM in total.
- VM1: min 4 GB RAM, max 14 GB RAM, Ballooning: ON
- VM2: min 4 GB RAM, max 14 GB RAM, Ballooning: ON
- VM3: min 4 GB RAM, max 14 GB RAM, Ballooning: ON
---------
VM min total: 12 GB RAM out of 32 GB RAM in node
VM max total: 42 GB RAM out of 32 GB RAM in node

When I turn on the VM's, 1 is always turned off. I understand that this is from lack of RAM, but that's why there is MIN RAM and Ballooning.

So why is this mechanism not working? And how to make it work? Many thanks!

 

[Dunuin]

Any PCI passthrough involved?

 

[corndelphinus]

Any PCI passthrough involved?

Hello! In VM settings -> Hardware, there is no "PCI Device" and i don't have any at host.

 

[corndelphinus]

I tried it with only 1 VM (8 GB min, 26.4 GB max RAM).

It looks like ballooning is taking 1.5 hour to take effect. (node RAM 31.09 GB in total, 80 % is 24.9 GB for VM's)

This looks like it's not able to resolve RAM issues "real-time"

 

[Dunuin]

This looks like it's not able to resolve RAM issues "real-time"

It can't shink the VMs in real time. If it would immediately remove the RAM from the VM the guestOS would panic and wouldn't have time to swap out data. So it does this slowly. But 1.5 hours is way too long...

 

[corndelphinus]

It can't shink the VMs in real time. If it would immediately remove the RAM from the VM the guestOS would panic and wouldn't have time to swap out data. So it does this slowly. But 1.5 hours is way too long...

In that case, ram ballooning makes more sense on servers with huge ram size, where the 20% gap is really big and the server can balance more between VM's in longer periods of RAM, right?

Ballooning relies on the fact that, on average, there will not be a peak in RAM usage across all VMs because it cannot react quickly to such a situation and simply shuts down the VM.
Am I understanding this correctly?

 

[ALFi]

ballooning/ksm/cpu thin provisioning makes sense when you want to overprovision your resources. the more resources you have the more sense it can make.
e.g. wanting to save mem on an 8GB system, most likely it is not worth the trouble and smarter to get another memory stick.
but if you are an ISP and have hundreds of servers with +1TB of mem each. you can literally save millions if you do a little overprovisioning instead of buying hardware.

you now how it is, the IT asks the devs how much resources the new app will need (they overestimate a little), they ask product management how many millions of users are expected (they are always way overoptimistic), so usually you can get away with overprove in this bigger environments just fine.

for small systems i think containers are better if you want to get the most use out of your last bit of memory instead of using vm.


also the host can not "take" memory from the vm, he asks the vm if it can have back some memory and the VM decides if it will hand out memory or not.
this is usually not one single interaction, most likely they do many iterations and only a few kb/mb are returned each round, that is why it is slow.
(if the host runs out of free mem then it has to swap or oom some processes which usually are the biggest ones -> your kvm/qemu vms)

also the other way around if the vm wants it's dynamic memory back it has to ask the host for it, the host has to decide if it will give memory back or not. (if it does not get the mem fast enough, it has to swap or oom some processes)

to the question about the minimum size setting, there are programs that have their own memory scheduler and require the whole range when starting.
e.g. java: if you have the java app set to use 8GB and have a vm with currently 6GB free and an possible additional 4GB via balloon, the java app will crash while starting due to out of memory, ballooning in this case is not fast enough... see that happen all the time
in this cases the minimum setting makes sense to get right.


also right sizing your vm's is better then overprov, but really hard to get right..
we usually get away with 120% mem and 350% cpu overprov, but each environment is different.

and for some reason it rubs me the wrong way that it is called ballooning with kvm..
ballooning is from ESX where you have an process inside the vm that gets inflated (like a balloon, hence the name), those pages are then mapped out by the host. also this balloon is backed by it's own swap file, so on esxi a VM might get really slow in low mem conditions but will never be oom by the host.
on kvm/qemu the memory size of your vm will shrink and grow.. so dynamic/flexible memory would have been a much better name for this and if the host does not have enough swap space it may trigger an oom depending on how crazy you go with overprov.

 

[corndelphinus]

ballooning/ksm/cpu thin provisioning makes sense when you want to overprovision your resources. the more resources you have the more sense it can make.
e.g. wanting to save mem on an 8GB system, most likely it is not worth the trouble and smarter to get another memory stick.
but if you are an ISP and have hundreds of servers with +1TB of mem each. you can literally save millions if you do a little overprovisioning instead of buying hardware.

you now how it is, the IT asks the devs how much resources the new app will need (they overestimate a little), they ask product management how many millions of users are expected (they are always way overoptimistic), so usually you can get away with overprove in this bigger environments just fine.

for small systems i think containers are better if you want to get the most use out of your last bit of memory instead of using vm.


also the host can not "take" memory from the vm, he asks the vm if it can have back some memory and the VM decides if it will hand out memory or not.
this is usually not one single interaction, most likely they do many iterations and only a few kb/mb are returned each round, that is why it is slow.
(if the host runs out of free mem then it has to swap or oom some processes which usually are the biggest ones -> your kvm/qemu vms)

also the other way around if the vm wants it's dynamic memory back it has to ask the host for it, the host has to decide if it will give memory back or not. (if it does not get the mem fast enough, it has to swap or oom some processes)

to the question about the minimum size setting, there are programs that have their own memory scheduler and require the whole range when starting.
e.g. java: if you have the java app set to use 8GB and have a vm with currently 6GB free and an possible additional 4GB via balloon, the java app will crash while starting due to out of memory, ballooning in this case is not fast enough... see that happen all the time
in this cases the minimum setting makes sense to get right.


also right sizing your vm's is better then overprov, but really hard to get right..
we usually get away with 120% mem and 350% cpu overprov, but each environment is different.

and for some reason it rubs me the wrong way that it is called ballooning with kvm..
ballooning is from ESX where you have an process inside the vm that gets inflated (like a balloon, hence the name), those pages are then mapped out by the host. also this balloon is backed by it's own swap file, so on esxi a VM might get really slow in low mem conditions but will never be oom by the host.
on kvm/qemu the memory size of your vm will shrink and grow.. so dynamic/flexible memory would have been a much better name for this and if the host does not have enough swap space it may trigger an oom depending on how crazy you go with overprov.

Thank you for this detailed answer. Make sense. (y)

Unfortunately there is still a long way till 1 TB of RAM on server

 

[Dunuin]

also the host can not "take" memory from the vm, he asks the vm if it can have back some memory and the VM decides if it will hand out memory or not.

This is not how I understand that KVM ballooning works. Host simply slowly removes RAM from the VM, once the hosts RAM exceeds 80%, no matter if the guestOS actually needs that RAM or not. The guestOS then has to deal with it which also means killing important processes once no cache could be dropped.

 

[ALFi]

This is not how I understand that KVM ballooning works. Host simply slowly removes RAM from the VM, once the hosts RAM exceeds 80%, no matter if the guestOS actually needs that RAM or not.

hm.. well last time we testet it was like 3-4 years ago on ovirt/rhev.
there it looked that way, that the guest can refuse to let go of mem when it currently has onlyactive used mem.
i have not actually tested with proxmox.
the underlaying kvm/qemu should be the same.

after short google, redhat did some memory management with their mom and libvirt.
here it is covered/tested: https://medium.com/coccoc-engineering-blog/the-way-of-kvm-memory-management-9670f0f852af
at the end one can even see that it expands/shrinks only small amounts at a time.
and here an older test with mom and ballooning: https://developer.ibm.com/tutorials/l-overcommit-kvm-resources/
also ovirt has an cluster scheduler that moves vm`s to another host if resources get to low..

well then i am wrong with vanilla kvm/qemu, but isn`t there an comand or something in /proc where you can online manipulate the balloon and test this?

if vanilla kvm/libvirt is really writen in a way that it balloons without the vm's concent and the vm crashes due to that thats kind of dump and i would consider this an bug.

i can understand the host running out of mem/swap and then oom a vm.

The guestOS then has to deal with it which also means killing important processes once no cache could be dropped.

on the other hand one could argue that this vm is then missconfigured since it has not enough swap to handle such a situation

 

[fireon]

A snap from the Proxmox Documentation:

When setting the minimum memory lower than memory, Proxmox VE will make sure that the minimum amount you specified is always available to the VM, and if RAM usage on the host is below 80%, will dynamically add memory to the guest up to the maximum memory specified.
When the host is running low on RAM, the VM will then release some memory back to the host, swapping running processes if needed and starting the oom killer in last resort. The passing around of memory between host and guest is done via a special balloon kernel driver running inside the guest, which will grab or release memory pages from the host.
When multiple VMs use the autoallocate facility, it is possible to set a Shares coefficient which indicates the relative amount of the free host memory that each VM should take. Suppose for instance you have four VMs, three of them running an HTTP server and the last one is a database server. To cache more database blocks in the database server RAM, you would like to prioritize the database VM when spare RAM is available. For this you assign a Shares property of 3000 to the database VM, leaving the other VMs to the Shares default setting of 1000. The host server has 32GB of RAM, and is currently using 16GB, leaving 32 * 80/100 - 16 = 9GB RAM to be allocated to the VMs on top of their configured minimum memory amount. The database VM will benefit from 9 * 3000 / (3000 + 1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server from 1.5 GB.