Maximum useful Ram for ZFS

[mickier]

I have seen many posts on this topic but none seem to talk about very big servers... I am installing a couple proxmox servers with 256GB ram each. How much RAM can ZFS use? for best performance...
Almost every post seems to be asking the MINIMUM recommended - I'm asking the MAXIMUM that will still benefit my server...

Most posts say ZFS will try to use 1/2 my RAM. I'm kinda OK with it taking 128GB ram but only if it will actually help with performance... Otherwise I'd limit it to "only" 64GB or something.

Any advice? (ps, I have 2x 1.8SSD drives and 4x8TB SAS drives, total of~36TB storage, zfs Raid10) R730

 

[narrateourale]

If data that needs to be read is in the cache/RAM (ARC), then a trip down to the disks is avoided. If you have some monitoring, it can fetch stats on how often the ARC could not satisfy a read operation. arcstat

On some of my servers with 128 GiB or RAM, the ARC will vary in size from 40 to 62 GiB. The hit rate is usually in the high 90%. Most times 99.9xx%. When doing updates, installing software or accessing older, not often used data, that can drop of course.

Depending on the workload your servers will see, the results might not be that good.

Since the servers are using enterprise SSDs, the impact isn't that easy to spot anymore. But back when I had HDDs in it, the difference between the snappiness was definitely noticeable. Day to day the VMs were snappy and fast. Only during updates/installations you could feel that they were stored on slower HDDs.

 

[Dunuin]

No one can really answer that. The bigger the ARC the better. But to not waste money you best try different ARC sizes and monitor the hit rates (for example by running arc_summary) and then try to find the sweetspot for your hardware and workload where you get the most bang for the buck in terms of higher ARC hit rates vs higher RAM cost.

I personally usually start with the default 50% RAM and then lower it step by step until I see a bigger drop of the hit rates.