Deciding on L2ARC size (if at all)

[NewDude]

Situation: I'm evaluating a cluster built on low-end Dell T30 computers with ram maxed out at 64G. I have the recommended Samsung SSD as a SLOG using a 32G partition, and I'm trying to decide if I should dedicate the rest of the drive to L2ARC functionality, and if so how much of it?

In concept the L2ARC makes a ton of sense, but I remember warnings from a few years ago about not using it, or marginal gains at the risk of huge performance penalties. But (re-)reading the ZFS pages on pthree.org are making me think I've misunderstood something.

All that will be stored are VMs, but they all do database-backed web serving of one sort of another.

Is there a rule of thumb about cache sizing that most people agree on?

 

[denos]

L2ARC is often misunderstood. I'll answer your question but with a few extra details for Google's sake.

Having an L2ARC does eat into the RAM available for primary caching via the ARC. If the hot areas of your ZFS pool(s) would have just fit nicely into the ARC without an L2ARC, you will take a performance hit as some of that data will now be going to the L2ARC -- but only if was accessed via random IO, otherwise it's going to come off the disks.

Which brings us to: the L2ARC will only cache random IO activity (not sequential) by default. Sequential operations are not a problem for any kind of disk (even modern 7200RPM spinning disks can now do ~ 150+MB/s on a single disk) so they'll be fed directly from the underlying disks if the data isn't already in the ARC. You can override this but your SSD may have a very short life if you do. If you don't do a lot of random IO, the SSD won't help you much and you will be trading a chunk of RAM that the ARC could have used (it caches everything).

You can dynamically add and remove a cache device, so I suggest trying it. Give it at least a month before making a decision. The statistics for the L2ARC are regularly misinterpreted so here's an example to help you make an informed decision (look under the "L2 ARC Breakdown" heading):

# arc_summary.py -p 3

------------------------------------------------------------------------
ZFS Subsystem Report                Fri Mar 02 14:29:09 2018
L2 ARC Summary: (HEALTHY)
    Low Memory Aborts:            33
    Free on Write:                138.03k
    R/W Clashes:                13
    Bad Checksums:                0
    IO Errors:                0

L2 ARC Size: (Adaptive)                773.97    GiB
    Compressed:            46.48%    359.72    GiB
    Header Size:            0.07%    551.92    MiB

L2 ARC Evicts:
    Lock Retries:                293
    Upon Reading:                17

L2 ARC Breakdown:                794.82m
    Hit Ratio:            10.87%    86.39m
    Miss Ratio:            89.13%    708.43m
    Feeds:                    3.56m

L2 ARC Writes:
    Writes Sent:            100.00%    3.55m

A hit rate of only 10.87%? That's terrible right? That's the standard interpretation, but it's wrong. Remember the L2ARC is only helping us with random IOs which perform very poorly on spinning disks. Over a couple of months, this server has put 3.56m feeds into the L2ARC, but those 3.56m feeds have been hit 86.39m times. If that was an investment, you're looking at a 2427% rate of return. It's even better because those operations would have hurt performance on the server (observed as increased latency and decreased throughput due to IO wait).

Also, the misses reflect all IO (including sequential), not just L2ARC eligible random IO which is why it's nearly impossible to see a high L2ARC hit rate. The only time a high L2ARC hit rate would be expected is if nearly all IO was random.

If you do decide to give it a spin, you should increase the rate at which the L2ARC is allowed to feed data to your SSD as the defaults are too small (IMO). These changes allow up to 160MB/s to be written to the L2ARC (default is 16MB/s).

# Put this in /etc/modprobe.d/zfs.conf
options zfs l2arc_write_max=83886080
options zfs l2arc_write_boost=83886080

Then make it stick with the next reboot:

update-initramfs -u

I would give the L2ARC the remainder of that SSD. It will only use what it needs.