Memory management python in VM

[mmhx]

Hi there,

I'm looking for general advice on how to handle a VM with sporadic huge memory usage. In my case this is an ubuntu VM that wants a lot of memory for PIV calculations (or postprocessing, or something similar) in python.


The host has 256GiB available, and - first problem - when I started with something like minimum 32GiB and maximum 180GiB, the calculation would fail (with some memory related error) without ever getting close to the 180GB (according to proxmox GUI and to free -h which didn't even show all memory as either available or free only up to 96GiB or so). Why ? Could this literally have been 2 giant junks of

At that point, I read up on ballooning again and misunderstood it (that it would offer the guest more memory when the guest's memory was below 80% - when its the other way around, that the host is telling the ballon to hog memory from the guest when the hosts's memory is not below 80% usage) and tried filling the memory with ng-stress and finally just set the minimum to the maximum memory - which worked, but I obviously didn't want this one VM to block that much memory.

This is question 2: how is this adequately handled? Allow multiple VMs access to a lot of resources temporarily (overprovisioning) - without having the host kill them because it gets scared.


The scientist on the VM wanted even more, so we tried 230GiB maximum memory and 180GB minimum. At 14:46 the VM got OOM-terminated (found it in the host's syslog), weirdly this doesn't reflect in the guest's graph - only in the hosts.

(Memory usage of Guest)



Memory usage of Host

Question 3:
What % of RAM is safe to "allocate away" from the host? It doesn't run anything specific atm (only proxmox), so even 5% should be fine for it on itself. What are / where can I find the best practices for this ? And why did nothing get killed before when the memory usage was even higher on the host?



PS: I am also urging the scientist to check whether his calculation could be refactored or split or maybe use storage for caching or junks, I'm definitely also going to look into that, but the above questions are something I want to understand.

 

[Dunuin]

The host has 256GiB available, and - first problem - when I started with something like minimum 32GiB and maximum 180GiB, the calculation would fail (with some memory related error) without ever getting close to the 180GB (according to proxmox GUI and to free -h which didn't even show all memory as either available or free only up to 96GiB or so). Why ?

Thats how ballooning works. As soon as the hosts RAM usage exceeds 80% ballooning will kick in and slowly steal RAM from the guests until either:
A.) hosts RAM usage drops below 80% or
B.) all guests RAM got reduced to "Min RAM"and can't be reduced any further.
Ballooning doesn't care if that guest is needed that RAM or not. It will just steal it from the VM and the VM has to handle this. So the guest OS will first try to drop caches and if there is no more cached data to drop, it will start to kill your processes. If yu don't want ballooning to kill your python process, then you shouldn't set "Min RAM" lower than what that VM always atleast needs to to run the system and all your processes.

This is question 2: how is this adequately handled? Allow multiple VMs access to a lot of resources temporarily (overprovisioning) - without having the host kill them because it gets scared.

You can't really overprovision your RAM.

What % of RAM is safe to "allocate away" from the host?

That really depends. PVE itself will run fine with just 2GB of RAM. In case you use ZFS this also needs additional RAM for its ARC. And in case you use any cache mode like writeback this needs additional RAM too.

 

[mmhx]

B.) all guests RAM got reduced to "Min RAM"and can't be reduced any further.
Ballooning doesn't care if that guest is needed that RAM or not. It will just steal it from the VM and the VM has to handle this. So the guest OS will first try to drop caches and if there is no more cached data to drop, it will start to kill your processes. If yu don't want ballooning to kill your python process, then you shouldn't set "Min RAM" lower than what that VM always atleast needs to to run the system and all your processes.

That means since I set my "Min RAM" equal to "Max RAM", I skipped the step of processes getting killed in my VM and went directly to having processes in the host killed?
I think my other VMs were safe because they were all idle and within Min RAM specs...

Argh, I really thought there would be nice solutions for this :-/
In that case I hardly gain from virtualization, could as well have a physical machine and let people sign in and out of a paper list to organize who gets resources. (Okay I still have environments prepared and so on...)

Any idea why using all allocated memory failed in the beginning? I shouldn't have been ballooning, since I had way more free RAM on host and the RAM on guest also wasn't exhausted.
Oh another idea: would more swap inside the VM or on the host help me in

 

[Dunuin]

That means since I set my "Min RAM" equal to "Max RAM", I skipped the step of processes getting killed in my VM and went directly to having processes in the host killed?

With "Min RAM" = "Max RAM" ballooning won't remove RAM from the VM when the hosts needs it (hosts RAM >80%). If you then overprovisioned your RAM and your host runs out of RAM, it will first try to swap that RAM out. After your swap is full (or you got no swap), your host will start to kill processes to free up some RAM. Here it will usually kill the KVM processes first which are running your VMs. So yes, when your host runs out of RAM it will first kill you VMs and crash them.

 

[mmhx]

Yeah, so I could temporarily get this working by reducing the RAM again and adding a bunch of swap (SSD) - which is obviously not good for continued work.
I will look into zRAM and zSWAP, because lowering the speed is fine (nobody is waiting for "the calculation to finish" before continuing to work), and depending on how well the compression works I would actually increase the % to quite a high amount...
I can imagine ballooning and zRAM might not work well together.

 

[mmhx]

OK, what I experienced following this:
I was notified of reduced RAM inside the above-mentioned (memory-hogging) VM. I suspected ballooning. I saw that the RAM usage on the host was really high (above threshold for ballooning) and that other VMs were suddenly using increased amounts of RAM (in proxmox VE) in idle, and I couldn't find out (in top) what process it was.

My guess is that ballooning kicked in (should that show as a process in the guest?), but how does that show?
- is there just a difference between free and available memory (in Linux systems)?
- Or should it show as used memory (also inside the VM)
- with or
- without a process assigned ?

 

[Dunuin]

You can for example use free -h in linux guests. Not the used RAM of the guest will increase, the total RAM of the VM will decrease. Lets say you got a VM with "Max RAM = 4GB" and "Min RAM = 2GB". Then without ballooning active the VM got 4GB RAM. When ballooning kicks in the total RAM will slowly be reduced. 3.9GB, 3.8GB, 3.7GB, 3.6GB ... until the VM only got a total RAM of 2GB. The webUIs graphs won't show this. Lets say that VMs RAM got reduced from 4GB to 2GB and 1.9GB of that 2GB is "used" with just only 0.1GB "free". In reality, your VMs RAM is 95% full and only 5% free. But the PVE webUI will still show it as 1.9GB of 4.0GB, so 47.5%. RAM "used".

But when your VMs total RAM gets smaller, the "used" to "free" RAM ratio of cause gets worse, as the VM is still running the same workloads needing the same amount of RAM, but with less total RAM available and ballooning will steal the "free" RAM first. So higher percentage of RAM usage from the point of view of the guest OS.

 

[mmhx]

Thank you. I wasn't sure what I'd be looking for.
I think I'll monitor it a bit and then come back with more questions (probably)