[SOLVED] ZFS on SSD is slow

[showiproute]

Hello everyone,

I am currently using ZFS on my HDDs as well as on my SDDs.
On my HDDs I have recoginzed the "downside" of ZFS: If I copy a lot of data the server load and RAM usage goes up which is fine - also the throughput sometimes stop.

What I do not understand is the same behaviour on SSDs.

HDDs: I copy stuff with ~1 GB/s a couple of seconds, than the transmission goes to 0 and I have to wait. I think the data needs to be written from a Cache to the disk.

SSD: I copy stuff also with ~1 GB/s. Transmissions goes to 0 for a longer time and data are being witten with around ~200 MB/s. To be honest I am using a low end SSD (Samsung QVO 2TB) but I would expect a little higher transmission.

 

[Dunuin]

Most SSDs (especially QLC NAND) are extremly slow at writing as soon as the cache of the SSD is full. Your SSDs sequential writes should drop down to around 160MB/s as soon as RAM and SLC cache are full. For random writes it will be much slower and could be as less as some MB/s.
If you want to write much data at once you should look for SSDs with SLC/MLC and not TLC/QLC NAND.

 

[showiproute]

Okay so in general I'd look for a SLC SSD for enhanced performance.

Is there any Pro-sumer SSD you might recommend?

 

[Dunuin]

It isn't recommended to use consumer/pro-sumer SSD if you want to use Copy-on-Write file systems like ZFS because of the high write amplification. Especcially if you write a lot of data, what your usecase looks like if you want to write at 1 GB/s. I think today only enterprise grade SSDs are using SLC/MLC NAND, because that typ of NAND is too expensive. You could look for second hand enterprise grade SSDs. You get them with only 0-5% of TBW used (because the TBW is for example 30 times higher compared to consumer SSDs of the same size) for quite cheap. Just check the SMART values before buying.

 

[showiproute]

And do you also know if it's possible to use for caching SSD for multiple pools

 

[Dunuin]

You can use a SSD as L2ARC (read cache) but is only useful if you already maxed out your RAM and you can't add more. And you could use a SSD as a SLOG (write cache) but that only works for synchronous writes and most stuff like SMB shares uses asyncronous writes what isn't covered.
What could make sense is using a SSD as special device. That way small files like zfs metadata are stored on the SSD and not on the HDD pool. But make sure to mirror the SSD because if that SSD dies your complete pool is lost.

This is a nice article: https://arstechnica.com/information...01-understanding-zfs-storage-and-performance/