KVM killed by OOM killer - Out of memory (ZFS problem?) / Proxmox 8.1.4

[mhert]

Hello!

We bought a new backup server last year. The file system is ZFS. The memory usage is always high, which seems to be normal with zfs.





The server has 64GB RAM and there is one virtual machine with 16GB RAM (Windows Server 2019). So after one day (since a reboot) the memory usage of the host is
nearly 90%. Strange but no problem so far.


Last week I installed another VM with 8GB RAM (Server 2019) for testing purposes. This morning I wanted to connect to the machine (RDP) but it wasn't reachable, also the "older" VM
on the same host.

I rebooted the host by IPMI and after everything works fine again, I found this in the logs:

Jan 27 15:10:41 prox11 kernel: Out of memory: Killed process 4299 (kvm) total-vm:24478756kB, anon-rss:16856924kB, file-rss:1792kB, shmem-rss:0kB, UID:0 pgtables:45700kB oom_score_adj:0
Jan 27 15:10:41 prox11 systemd[1]: 101.scope: A process of this unit has been killed by the OOM killer.

Is there something wrong with the ZFS-Configuration (standard installation)?

Detail view: (similar output at the other pools)




Thanks in advance, itiser

 

[leesteken]

We bought a new backup server last year. The file system is ZFS. The memory usage is always high, which seems to be normal with zfs.

Yes: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_limit_memory_usage

The server has 64GB RAM and there is one virtual machine with 16GB RAM (Windows Server 2019). So after one day (since a reboot) the memory usage of the host is
nearly 90%. Strange but no problem so far.

ZFS takes 32GB, the VM 16.5GB, Proxmox takes 2GB, which adds up to 80% or so. Maybe the rest is filesystem cache for other filesystems besides ZFS?

Last week I installed another VM with 8GB RAM (Server 2019) for testing purposes. This morning I wanted to connect to the machine (RDP) but it wasn't reachable, also the "older" VM
on the same host.

I rebooted the host by IPMI and after everything works fine again, I found this in the logs:

Jan 27 15:10:41 prox11 kernel: Out of memory: Killed process 4299 (kvm) total-vm:24478756kB, anon-rss:16856924kB, file-rss:1792kB, shmem-rss:0kB, UID:0 pgtables:45700kB oom_score_adj:0
Jan 27 15:10:41 prox11 systemd[1]: 101.scope: A process of this unit has been killed by the OOM killer.

OOM usually kills the process that uses the most memory, which is your 16GB VM. I don't what triggered it specifically.

Is there something wrong with the ZFS-Configuration (standard installation)?

No, but the pool was created by an earlier version of ZFS with less features.

 

[mhert]

Is there no control function to lower the memory usage of zfs if there are other processes which need memory?

 

[LnxBil]

Is there no control function to lower the memory usage of zfs if there are other processes which need memory?

Of course there is, but sometimes it is too slow.

The problem could also be memory fragmentation.

 

[mhert]

How do I limit the size of memory for zfs, e.q 24GB ?

 

[LnxBil]

How do I limit the size of memory for zfs, e.q 24GB ?

As @leesteken already pointed out in #2:

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_limit_memory_usage

 

[mhert]

Thanks. I've overlooked that... ;-)

 

[mhert]

I followed the guide und added the following line (24GB)

cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=25769803776

The result is an entry (after reboot and "update-initramfs -u -k all") in:

cat /sys/module/zfs/parameters/zfs_arc_max
25769803776

The UI shows:



Actually the ram usage should not be much more than 44 GB... (ZFS_24GB + VM1_16GB + VM2_4GB (I lowered that for testing purposes) + X for Proxmox). Or is it OK (if it does not continue to rise)?

 

[LnxBil]

A VM needs more memory than you assign to it. There is also virtualized hardware that needs ram, also disk buffers / cache etc.

The easiest solution would be to add more RAM. 64 GB is not much in virtualization and was not 10 years ago.

 

[mhert]

It seems to work... till now... ;-)

Thank you for your help!