zfs l2arc & secondarycache=metadata instead of special vdev - worth a try ?

[RolandK]

is anybody using l2arc on single cheap consumer ssd with pbs, instead of using special device mirror (metadata allocation class) , as recommended in the pbs documentation - and would like to share his experience with that ?

i did not do exact performance comparison before/after yet, but my impression is that it's a viable and safe method to speed things up if your backup pool is on hdd.

 

[Dunuin]

I've read multiple users here are using L2ARC + secondarycache=metadata instead of the special vdev so they don't need redundancy and would be able to remove it without destroying the complete pool. But cheap consumer SSDs are never recommended with ZFS, neither as an cache disk nor a data disk.

 

[RolandK]

>But cheap consumer SSDs are never recommended with ZFS, neither as an cache disk nor a data disk.

yes, i would agree, but i where do you see a problem why they wouldn't be ok for metadata only caching (as this is far far less read/write operations then with secondarycache=all) ?

 

[Dunuin]

Bad performance in general and especially for sync writes (a 15K HDD can be faster than a consumer SSD depending on the workload), less reliable, worse monitoring capabilities, very bad life expectation, no powerloss protection so higher chance of corrupting stuff, ... also keep in mind that persistent L2ARC is still an experimental feature that can cause problems. Because of that stuff like TrueNAS isn't using that yet. So depending on how often you reboot your PBS server writes can add up quickly because that L2ARC will be wiped and rewritten on each reboot. But atleast it is only used as a read cache, so you won't loose data if that SSD will suddenly fail and PLP shouldn't be that important.

And also keep in mind that L2ARC will use additional RAM, so your ARC will get smaller. The bigger your L2ARC the more additinal RAM you need to buy.

 

[RolandK]

>... also keep in mind that persistent L2ARC is still an experimental feature that can cause problems

hu ? l2arc_rebuild_enabled defaults to "yes" , so where is the warning sign or where is that info from? do you have a pointer ?

>And also keep in mind that L2ARC will use additional RAM, so your ARC will get smaller. The bigger your L2ARC the more additinal
>RAM you need to buy.

sure, but is that really relevant/significant for some tens of gigabytes of metadata ?

 

[Dunuin]

>... also keep in mind that persistent L2ARC is still an experimental feature that can cause problems

hu ? l2arc_rebuild_enabled defaults to "yes" , so where is the warning sign or where is that info from? do you have a pointer ?

Latest TrueNAS Core Release Notes under "Known Issues":

NAS-106992

Persistent L2ARC is disabled by default.

While the underlying issues have been fixed, this setting continues to be disabled by default for additional performance investigation. To manually reactivate persistent L2ARC, log in to the TrueNAS Web Interface, go to System > Tunables, and add a new tunable with these values: Type = sysctl Variable = vfs.zfs.l2arc.rebuild_enabled Value = 1

>And also keep in mind that L2ARC will use additional RAM, so your ARC will get smaller. The bigger your L2ARC the more additinal
>RAM you need to buy.

sure, but is that really relevant/significant for some tens of gigabytes of metadata ?

Atleast for TrueNAS the documentation recommends not to use more than 10 times the ARC size for your L2ARC. So if you for example would use a 1TB SSD that would mean you want atleast a 100GB ARC. Thats alot of RAM.

 

[RolandK]

thanks for sharing.