[SOLVED] VM get killed by OOM Killer

[plop28]

Hello Guys,

I'm new with proxmox. I have installed latest PVE on a small server with 32GB of physical ram (swap disabled during tests)
I have seen many thread about this but I can't manage to get the ballooning memory working

I have two guests (debian11) qemu-guest-agent installed and active, if I go to the monitoring tab i can see that balloon driver is working on guests (info balloon)
My two debian guest have both 20GB of ram (max value 20480, min value 2048, 'ballooning device' is checked)

Now if i run a test with stress-ng on first guest machine, it consumes the ram entirely.
then I stop stress-ng on the guest. I can see with free -m that i have 547Mo used and 18937 free

Now i start the second guest, and try to do the same thing. Ram is growing on the node to reach 100%, then OOM kills the first machine.

Ballooning memory as I understand should be able to reclaim unused memory on my first guest and share it to the second ?

thanks for your help !

 

[plop28]

anyone ?

I guess I don't use ZFS as pool list is empty. i'm with lvm-thin

 

[dcsapak]

Ballooning memory as I understand should be able to reclaim unused memory on my first guest and share it to the second ?

Yes, it should, but this is not infinitely fast. If you start stress-ng, it will basically instantly try to use the available ram, which can lead to OOM situations like this.
If you want to be on the safe side, do not over commit your memory. If you do, there is always a chance that both vm need that memory at the same time.

Also note that if you are using pci-passthrough (idk if this is applicable here) the vm will always be using the whole assigned memory.

 

[plop28]

Hello, thanks for your reply !

I'm not using pci-passthrough for my test. I'm looking for replace my ESXis at work and I wanted to try the feature which btw works well on vsphere.
Is there some tuning on proxmox about this ? If the feature is not fast, then it is not usable.

Today test : RAM 16GB VM1 / 17GB VM2
16GB consumed on VM1. I stop the test. 1 hour later i consummed 17GB on VM2 and this lead me to the same problem. why the VM2 is not able to take 1GB of free VM1 RAM which has plently of free RAM during a long time ? Why on the node RAM graph, 50% of the RAM is still used after 1 hour of idle time ?
If i use swap partition, why guest is taking the 1GB on swap instead of use available ram on guests ?

Are you saying that if an app increases ram consumption very slowly, the feature will work then ?

 

[Dunuin]

Today test : RAM 16GB VM1 / 17GB VM2
16GB consumed on VM1. I stop the test. 1 hour later i consummed 17GB on VM2 and this lead me to the same problem.
why the VM2 is not able to take 1GB of free VM1 RAM which has plently of free RAM during a long time ?

Same here but nobody answer how that is supposed to be. Doesn't look like KVM is freeing up RAM. So if VM1 will go from 0 to 16 GB RAM usage the RAM used by the KVM process will increase from 0 to 16GB (or even more like 17 or 18GB because of overhead) too. But if the guest frees up memory (and I mean really "free" not "available") KVM won't give the RAM back to the host. The KVM process will still stay at 16GB (or 17-18GB) RAM until you stop that VM. So VM2 can't use 17GB because VM1 is still using the full 16 (or 17-18) GB. You could install htop and run that to see how much RAM your KVM processes are using.

Why on the node RAM graph, 50% of the RAM is still used after 1 hour of idle time ?

I guess you are using ZFS. ZFSs ARC will use up to 50% of your hosts RAM for caching. So its normal that your ram is 50% used when ideling without a VM running.

 

[dcsapak]

50% of the RAM is still used after 1 hour of idle time ?

the auto-ballooning only sets in when the host memory usage is >= 80% (see the Automatic Memory Allocation section of https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_virtual_machines_settings )

 

[plop28]

@Dunuin
I don't use ZFS (i guess) i don't have any pool configured

@dcsapak
yes I know that thanks. Then why ballooning doesn't work when the ram is at 80%, then 90, then 100% (then pve kills my VM)

 

[dcsapak]

again if you start stress-ng the memory usage is nearly instant, the auto-ballooning cannot keep up (neither can any other non-kernel mechanism for all such cases)
it can only update the ballooning ever so often

as i said, if you do not want to risk an oom situation, do not overcommit your memory

 

[plop28]

@dcsapak
is saw this response already, but i have had other related questions earlier in this thread :

Are you saying that if an app increases ram consumption very slowly, the feature will work then ?

thanks

 

[dcsapak]

Are you saying that if an app increases ram consumption very slowly, the feature will work then ?

yes

 

[plop28]

Ok i will do some further tests and i'll put some info here when done. thanks for your help @dcsapak

 

[Dunuin]

Ballooning really isn't a solution against OOM because of RAM overprovisioning. The host will use ballooning to lower the hosts RAM utilization so no VMs need to be killed by OOM but it is doing that by stealing RAM from the VMs so potencially forcing the guests OOM to kill the guests processes.
Ballooning will just lower the RAM without knowing if the VM could handle that. So first the VM will try to drop caches and if that isn't enough it will kill services too so that everything will fit in the reduced RAM size.

 

[plop28]

I'll don't do overprovisionning but I have a small hosted server with soft raid (that's why i'm trying proxmox over vmware) with small amount of RAM (32GB) There are some important servers there and I don't want one of them to be killed that way for some RAM MB if I'm doing a small task that requires me to consume more RAM than usual for a short period. It could happens

 

[plop28]

I'm back
I had some time recently to make tests, So yes, it works !

Same process on my host with 32GB of physical RAM:
Start VM1, run stress-ng to take 100% (20GB) of RAM, stop stress-ng
Start VM2, run a command that take slowly 18GB of ram by 10Mb/s (command : #</dev/zero head -c 18000m | pv -L 10m | tail)

Wait a while and the KSM sharing starts to increase :



Thanks for your help @dcsapak
Now I have another problem with apache reverse proxy and websocket connection but i need to read more about it before requesting help again