SLOG and L2ARC to speed up ZFS on a spinning drive

[Astaoth]

Hi,

I'm running a PVE cluster at home, mostly for fun and some local services. I deployed it on top of a couple of dell optiplex (for the low power consumption), which have 1 NVME slot and 1 SATA slot.

Initially, I put one 1TB NVME SSD inside both of the optiplex, partitionned with ZFS, and 16 GB of RAM. In order to have some HA, I have scheduled the VM and LXC replications every minute to every 15 minutes, depending of the services running inside each of them. It works well (especially with an additionnal corosync node running on a RPI).

But now, I'm thinking of deploying some new services which may need many local storage. I have found some 5TB spinning drives (5400 RPM) which can fit inside the dell optiplex (with some mod of the sata tray), but I fear I'll have some performance issues with the VM/LXC replications. I'm considering few different options, and one of them is to use the SSD NVME as a SLOG and L2ARC device. I have read few interresting stuff here and there on this topic, but most of them are from 2011 or 2015. So here are my questions :
- I know that having more memory is more usefull than using L2ARC, but when it will be the case (32GB max), will L2ARC be usefull in this kind of setup ?
- same question about SLOG : will it be of any help to solve the performance issues I'll encounter ?
- I have read that having both SLOG and L2ARC on the same device will produce some performance issue, so which one of them will improve the most the drive performances ?
- is all of this a wrong idea and should I consider an other solution instead ?

Thank you

 

[chrcoluk]

I plan to do some testing on this very soon, I now have a local system with spindles alongside SSD's.

To answer your question (before I do my testing), is it depends, which is your biggest bottleneck, reads or sync writes?

To improve read performance you cache the data, in addition a well tuned prefetch setup can also improve reads significantly.

So random reads, helped mostly by cache. sequential reads helped by both cache and prefetch. prefetch can hurt random reads.

By default reads are cached in ram (ARC), the size of the ARC, prefetch configuration, pattern of reads, and configuration can all affect the hit rate.

If your hit rate is low due to evictions adding a L2ARC can help improve that as evicted data is moved into the L2ARC device, it wont be as fast as ram but will be faster than sending the requests back to the main spindle storage. L2ARC is tuned for random reads out of the box but can be tuned to also work on sequential data. The common advice is to always bump up your ram to the max before using L2ARC, but I am not convinced by that advise (which is why I plan to test more), a smaller ARC means the L2ARC should populate quicker and now on openzfs 2.0.x the L2ARC survives reboots.

SLOG may improve random writes, by default async writes dont go to the SLOG, they instead go direct to the storage when the dirty cache is flushed, however if you feel async writes are causing you a problem, you can add a fast SLOG and set sync to always, to force all writes to the SLOG, however this also has the affect that the dirty cache wont be used with all writes been issued immediately. Bear in mind writes to the SLOG still have to be written eventually to the main storage.

As for using one fast ssd for both SLOG and L2ARC, I have little to comment on this, potentially if the workload is right it could work well, but if you have a very heavy sync write workload which keeps the device constantly busy and then its also writing evicted ARC data, you could overwhelm the device and cause a bottleneck. It would be for you to experiment I think, if it doesnt work out you can back out of the changes.

I will do the testing as all my production machines use spindles, it seems me and you are a rare breed with the majority using ssds for main storage now. This is probably why you have struggled to find information as most people just use $$$ to improve performance.