Unless @terry5008 buys QLC SSDs, then the HDDs will be faster on writesBut no matter what you do, your HDDs will never come close to SSD performance.
I thought creating a SLOG and L2ARC on SSD was the special device for metadata. What are you referring to when you say "special device"?What helps best are special devices to store the metadata, so all those small random reads+writes won't hit the slow HDDs.
No. SLOG is sync write cache. L2ARC is an additional layer of way slower but bigger read cache which also reduces your RAM so your fast ARC will shrink. You basically trade few super fast RAM read cache for more but slow SSD read cache. Depending on the workload this could even make your pool slower.I thought creating a SLOG and L2ARC on SSD was the special device for metadata.
https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954What are you referring to when you say "special device"?
Most of that explanation might as well have been written in ancient Greek.
Very gereralized, yes. HDDs suck at all random reads and sync writes. Many of those come from metadata so its a good thing to not have them on the spinning rust.Most of that explanation might as well have been written in ancient Greek.
So, the special class performance optimizing vdevs that can be added to a pool are as follows:
L2ARC = Basically a read cache on SSD(not terribly helpful because RAM is better)
SLOG = Which relocates the ZIL to SSD(only good for DB traffic)
Special = Metadata(and maybe small files) on SSD.
And it is your contention that the only one of these that will make VMs on spinning rust perform better is "special".
Correct?
When experimenting with different devices for use with SLOG and metadata, do these devices need to be taken offline before being removed from the pool? Or does the "zpool remove" command do this automatically?I can recommend to use a small (16 GB) Intel Optane for SLOG which has the most IOPS and WILL significantly increase all sync write in your environment. The point where it shines the most ist running any system update with e.g. apt, which uses a lot of sync write and sync calls and you will "see and feel" the speed, especially with harddisks. Yet besides those points, you will not see sooo much of them. Monitor your SLOG usage (e.g. with dstat) you will see how much or more precisely how little you write to your SLOG in general. For general harddisk speed, it can be fast. I use 22 disks over 3 SAS HBA controllers (6 channels) in a stripped mirror setup with two enterprise SSDs as metadata (special device) and one optane (I don't care about my last 5s of data in case of a loss). Bulk load maxes out at 2,5 GB/s, which is pretty fast. Random I/O is of course slower, but also good enough for most things.
I also tried L2ARC and it is not faster than special device in my book, not even close.
Is SSD shredding an issue if you only have about 3% of the SSD partitioned(massive overprovisioning)?Yes, useing partitions as special vdevs will work. just keep in mind that special vdevs are not cache. Lose them and all data on those HDDs will be lost. So you usually want the same redundancy/reliability as your normal vdevs (your HDDs) and I wouldn't share a SSD with SLOG and special vdevs as the SLOG really shreds SSDs.
a remove should be sufficient (it is for other types), yet I haven't done it yet. If you're not sure and want to 100% save, stop all applications reading/writing from/to it.When experimenting with different devices for use with SLOG and metadata, do these devices need to be taken offline before being removed from the pool? Or does the "zpool remove" command do this automatically?
Is SSD shredding an issue if you only have about 3% of the SSD partitioned(massive overprovisioning)?