KVM allocate memory failure - buff/cache not released

[adrien.islog]

Hi,

My proxmox have 47G Mem with 30G swap. It is using a local ZFS pool.
After a time, the buff/cache of the node totally use all the free memory (unused by current VM) and makes impossible to start a new VM or run a backup. From what I understood, backup is the same as starting a VM.
The VM start fail with: failed to initialize KVM: Cannot allocate memory

> free -h
total used free shared buff/cache available
Mem: 47G 33G 441M 63M 13G 13G
Swap: 30G 1.6G 29G

To run again a VM, I need to force drop cache with the command:
>sync; echo 3 > /proc/sys/vm/drop_caches

Here is the result:
>free -h
total used free shared buff/cache available
Mem: 47G 33G 13G 63M 205M 13G
Swap: 30G 1.6G 29G


What I do not understand is why I have to manually force to drop the cache ? do the kernel cannot itself clear the cache to let KVM allocate new mem ?

 

[spaun999]

Hi have same issue

can't move virtual machine "
kvm: failed to initialize KVM: Cannot allocate memory"

already make limit (Limit ZFS Memory Usage)

doesn't solve the problem

 

[spirit]

do you use cache=none (default), for vm disk option ?

and do you use zvol for vm disks ? (not raw on top of zfs)

 

[adrien.islog]

Hi @spirit,

Thank you for your answer.

Yes, I had write back cache enabled on Windows VM. I have changed to cache=none as you proposed.
The Windows VM are used as CI, so lots of build/write. It is possible that it is the reason of the cache size.

Otherwise all you VM disks use zvol.

I will get back with the result of the changes.

 

[adrien.islog]

Hi @spirit,

Unfortunatelly even after changing the cache to none, the buff/cache again reached a high number and no longer possible to boot a new vm even if there is free memory.
total used free shared buff/cache available
Mem: 47G 22G 458M 80M 24G 24G
Swap: 30G 4.1G 26G

Anyone else has an idea here ?

 

[sahostking]

Hi

I have the same issues and its seem to be happening on more nodes now.

I have to keep running this to get them to start up now:

echo 1 > /proc/sys/vm/compact_memory

 

[nilpferd]

Hi! Unfortunately I confirm the issue. 32 GB (dual socket, 16 GB each node) with 18GB free and I can only start a new VM after (manually) dropping the caches. compact_memory did not work for me. Disk cache is set to none.
Current ugly workaround is to drop caches from cron...