Correct ZFS memory management

[frankz]

Hello everyone, after following the posts and the official guide on how to limit the ZFS cache (arcstat), I was asking for your opinion regarding the configuration of my cluster.
The first node has 32GB of physical memory, and disks including the operating system in ZFS, a 1TB ZFS disk. To this node I gave 1/4 of the maximum memory or 8GB.
The second node has 16GB of memory, a 1TB and a 60GB ZFS disk. I gave a 1/4 of the memory or 4GB maximum.
The third node has 32 GB of memory a 1Tb ZFS disk and 1/4 of maximum memory or 8GB.
Do you think I can still file something or is it better to leave everything like this? Thank you

By carrying out this configuration, the cluster no longer goes into memory saturation, leaving several GB free for VM or LXC use

 

[Dunuin]

You can set a min and max for the arc size. If a VM needs more RAM, the ARC should free up that much RAM as long as ARC wouldn't be smaller than zfs_arc_min. So it should be fine to use something like zfs_arc_max = 16GB and zfs_arc_min 2GB. That way your ARC will always be something between 2 and 16GB and if a VM needs some of that RAM the ARC can shrink down to 2GB.
But if you are only using 1TB of storage a zfs_arc_max of 4GB should be totally fine for all 3 nodes.

 

[frankz]

You can set a min and max for the arc size. If a VM needs more RAM, the ARC should free up that much RAM as long as ARC wouldn't be smaller than zfs_arc_min. So it should be fine to use something like zfs_arc_max = 16GB and zfs_arc_min 2GB. That way your ARC will always be something between 2 and 16GB and if a VM needs some of that RAM the ARC can shrink down to 2GB.
But if you are only using 1TB of storage a zfs_arc_max of 4GB should be totally fine for all 3 nodes.

Thanks for replying. I know you can enter zfs arc min variable, but I wanted to follow the official site guide. I have participated in some posts on the subject and several users use as you say this variable. I noticed that if the VMs need memory the cache decreases, but I assume that from what you say it could reach 0 as the minimum variable has not been defined. I deduce that the consequence is a drop in performance of the ZFS without causing any stall. What do you think about it ?

 

[Dunuin]

You can test different ARC sizes and run arc_summary in cli. The bigger your ARC is the better, but if your hit rates are already high and dnode/metadata cache got enough room, adding additinal RAM to the ARC won't give you that much more performance.