ZFS L2ARC

[LnxBil]

Hi everyone,

This is strictly not a virtualization question, but a ZFS question, so I think that it'll be interesting for everyone involved with ZFS.

We have our ZFS-based backup server, which receives e.g. all dumps from our PVE cluster but also from other machines. The server is a HP DL 360G6, 72 GB-RAM, 96 GB L2ARC and three RAIDz2 vdevs consisting of 6 disks each, of which some are encrypted (others are in transition to be encrypted with LUKS).

root@backup ~ > zpool status -v rpool
  pool: rpool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 144h0m with 0 errors on Sat Dec 15 00:24:28 2018
config:

        NAME              STATE     READ WRITE CKSUM
        rpool             ONLINE       0     0     0
          raidz2-0        ONLINE       0     0     0
            sdh           ONLINE       0     0     0
            sdm           ONLINE       0     0     0
            R2Z3S3_crypt  ONLINE       0     0     0
            sdj           ONLINE       0     0     0
            sdl2          ONLINE       0     0     0
            R2Z2S1_crypt  ONLINE       0     0     0
          raidz2-1        ONLINE       0     0     0
            sdn           ONLINE       0     0     0
            sdo1          ONLINE       0     0     0
            R2Z1S2_crypt  ONLINE       0     0     0
            R2Z1S4_crypt  ONLINE       0     0     0
            sdr           ONLINE       0     0     0
            sds           ONLINE       0     0     0
          raidz2-2        ONLINE       0     0     0
            R1Z1S2_crypt  ONLINE       0     0     0
            R1Z1S3_crypt  ONLINE       0     0     0
            R1Z2S2_crypt  ONLINE       0     0     0
            R1Z2S3_crypt  ONLINE       0     0     0
            R1Z3S2_crypt  ONLINE       0     0     0
            R1Z3S3_crypt  ONLINE       0     0     0
        cache
          sdb1            ONLINE       0     0     0

errors: No known data errors

Everything is fine and we monitor including performance metrics and here is the statistics from ARC and L2ARC. I am a little surprised. In our setup and use case, we almost never use L2ARC:




We have not tuned anything specially, so I'd like to know about experience of others regarding this.

In the near future, we will expand the main memory and drop the L2ARC completely.

 

[fabian]

what about usage (e.g. check arc_summary.py)? sounds like most of your data fits well into ARC (~80% hit rate), which means that the L2ARC is not used (much). you need to read stuff from disk from time to time as well even if the caches work well - those obviously will be misses for both ARC and L2ARC.

 

[LnxBil]

what about usage (e.g. check arc_summary.py)?

L2ARC has 12 MB usage. That is not that much:

root@backup ~ > arc_summary -p 3

------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Jan 04 19:30:35 2019
L2 ARC Summary: (HEALTHY)
        Low Memory Aborts:                      138.00k
        Free on Write:                          45.75M
        R/W Clashes:                            449
        Bad Checksums:                          0
        IO Errors:                              0

L2 ARC Size: (Adaptive)                         106.01  GiB
        Compressed:                     90.49%  95.93   GiB
        Header Size:                    1.83%   1.93    GiB

L2 ARC Evicts:
        Lock Retries:                           262.69k
        Upon Reading:                           19.99k

L2 ARC Breakdown:                               199.12G
        Hit Ratio:                      0.06%   116.80M
        Miss Ratio:                     99.94%  199.00G
        Feeds:                                  31.12M

L2 ARC Writes:
        Writes Sent:                    100.00% 12.20M

 

[fabian]

could you post stats for both ARC and L2ARC (from the same point in time)? as I said, if the ARC already serves most requests (because your working set fits inside, or mostly does), L2ARC does not offer much benefit. see https://github.com/zfsonlinux/zfs/blob/zfs-0.7-release/module/zfs/arc.c#L6738 for some more low-level details and pointers on module parameter knobs that allow tweaking.

 

[LnxBil]

see https://github.com/zfsonlinux/zfs/blob/zfs-0.7-release/module/zfs/arc.c#L6738 for some more low-level details and pointers on module parameter knobs that allow tweaking.

Wow. That is a nice documentation what's going on there. Thank you.

could you post stats for both ARC and L2ARC (from the same point in time)? as I said, if the ARC already serves most requests (because your working set fits inside, or mostly does), L2ARC does not offer much benefit.

That is exactly what I see. The cache misses on the L2ARC are also present on the ARC, but much more is read from the ARC in terms of cache hits. The L2ARC is therefore not really necessary and we're going to remove it soon. Yesterday, we more than doubled the amount of RAM, so we're going to be fine.

 

[fabian]

it's probably also worth noting that L2ARC is not used for sequential reading (on the assumption that such reads are served fast from spinning rust as well, and the limited cache space is better used for random reads). more RAM is definitely a good thing in any case

 

[guletz]

Hi,

It is absolute normal, on a backup server, to have no L2arc use, because mostly, you will write on the pool(from your backup tasks), and the read's are very low(in normal usage case, when you do not make some backup restores).

Even if you run a restore, if you do not make it in short time after your bakap task was finished, the L2ARC will not be usaged. So for this resons, on zfs backup server, I do nu have any l2arc, and I set for arc to have cache only for metadata. If you want to keep l2arc(maybe you also have some others tasks then backups), you could set the same proprety for l2arc(metadata only)!

best regards,