Is it OK to have more than 90% used RAM with Proxmox VE on ZFS ?

[jhr]

I am trying to find the best config for my Proxmox on ZFS rpool. I know that ZFS eats so many RAM for caching.
When I am not limiting amount of arc in /sys/module/zfs/parameters/zfs_arc_max, sooner or later I have more than 90% of RAM used.
I have for example 64GB physical RAM, my one only VM running with 32GB RAM, so the rest is for Proxmox and ZFS.
When I setup some limits to zfs_arc_max, of course my IO going worst and backup of VM takes a much longer.
I cannot add more RAM (this MB has max 64GB), I cannot downsize my VM's RAM.
I am a little bit nervous when I see how much RAM is used and wondering if it is OK or not, cos all monitoring tools (nagios, checkmk) says that is critical.

Should I limit arc_max despite to worst IO or leave it alone as is ?

PS: I have added some swap to my PVE to be sure, that OOM will not kill my server. And of course sooner or later PVE eats some parts of swap too.

 

[billy999]

I have a 64GB server with 10GB for VMs and 50GB max arc size and Ive had 0 issues with it using around 93% of RAM. Simply leave enough headroom for your software running on host

 

[UdoB]

Should I limit arc_max despite to worst IO or leave it alone as is ?

It is fine as long as OOM killer is not triggered in your specific setup and use case.

Above 80% memory ballooning kicks in. For this reason I try to stay below "80 % used" on the host...

Edit: regarding ZFS ARC: yes, it is dynamic and it will shrink "automatically". The problem is that this needs some time. For a process/VM in need of RAM now! this often is too slow --> killing the process holding the most RAM will happen...

 

[RolandK]

it's typically not a problem that zfs proxmox hosts show high memory load.

it's not zfs arc's fault, the problem is that arc is a special case for reclaimable memory and many monitoring tools or even proxmox itself does not take into consideration that arc cache is special regarding memory utilization.

so you may please vote for

https://ideas.checkmk.com/suggestions/631042/make-checkmk-memoryram-monitoring-zfs-arc-aware

and

https://bugzilla.proxmox.com/show_bug.cgi?id=1454

and

https://bugzilla.proxmox.com/show_bug.cgi?id=3859

and

https://bugzilla.proxmox.com/show_bug.cgi?id=4482

 

[leesteken]

Above 80% memory ballooning kicks in. For this reason I try to stay below "80 % used" on the host...

Fortunately, this threshold is configurable since PVE 8.4: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#ballooning-target (also via the web GUI).