Speed up Backupserver by adding zfs special device ?

[UdoB]

Could i add two 500G consumer ssd in mirrored raid-z ?

You probably know the answer, technically yes. But no one will recommend that...

If you need to use consumer grade SSDs: 1) use different brands. 2) use "too large ones" intentionally. 3) run a mirror vdev consisting of three devices instead of just two. 4) make backups to physically separate devices (a PBS) more often than usual.

PS: "mirrored raid-z" makes no sense. "Mirrored" describes multiple (usually identical) devices which contain the exact same content. "RaidZ = RaidZ1"/RaidZ2"/"RaidZ3" describes a construct which distributes the data plus one (two/three) sets of parity data onto several devices.

 

[IsThisThingOn]

I agree on the different brands or even better use different controllers. Lots of consumer drives use the same Phison controller.
And of course 3 way mirror is always better than a 2 way mirror. Still think 2 way mirror is perfectly fine for my homelab.

What I don't understand is this part:

- you need fast and high-endurance SSDs (or preferably, NVME devices). no consumer/prosumer HW
- if your special vdev mirror is toast, the whole pool is gone

If you don't set it to store small files and leave it at the default to only store metadata, there is basically no TBW, so even a cheap SSD will do.
You also don't need bandwidth, so even a cheap old SATA3 SSD will do.
And "fast" probably mostly means high random 4k reads, which the cheapest SSD mirror will still be lightyears ahead of a RAIDZ HDD pool.

IMHO, special vdev is the least demanding application there is and you can use the crappiest SSDs you can find and still get a huge metadata performance boost.

 

[UdoB]

What I don't understand is this part:

If you don't set it to store small files and leave it at the default to only store metadata, there is basically no TBW, so even a cheap SSD will do.

For every single access to "data" at least one write to metadata is required. So the IOPS for the Special Device is really relevant.

You also don't need bandwidth, so even a cheap old SATA3 SSD will do.

Correct. Bandwidth is not the point here.

And "fast" probably mostly means high random 4k reads, which the cheapest SSD mirror will still be lightyears ahead of a RAIDZ HDD pool.

Maybe. I can not confirm that, but the suspicion is plausible.

Actually it would already help to add an Special Device build from HDDs - as it would separate Metadata from Data and double the IOPS of the pool. Nobody would recommend that...!

Anecdote as a sidenote: just yesterday I had cleaned up a box with used SSDs. One was a Samsung 750 EVO. Zeroing it with "dd" it was able to write data with 21 MB/s. Each and every harddisk based on rotating rust is multiple times faster. I did not test for IOPS.

 

[IsThisThingOn]

For every single access to "data" at least one write to metadata is required. So the IOPS for the Special Device is really relevant.

Hmm... that is right, forgot about that one. Still a very small writes.

Actually it would already help to add an Special Device build from HDDs - as it would separate Metadata from Data and double the IOPS of the pool. Nobody would recommend that...!

Probably because even the cheapest SSD outperforms any HDD, but yeah it would help.

Zeroing it with "dd" it was able to write data with 21 MB/s. Each and every harddisk based on rotating rust is multiple times faster. I did not test for IOPS.

You were basically testing sustained sequential writes. Extremely old SSDs and QLC SSDs offer worse numbers than a HDD when they ran out of pseudo cache. But on the IOPS front, even your 750 is way faster than any modern HDD.

 

[Fathi]

You probably know the answer, technically yes. But no one will recommend that...

If you need to use consumer grade SSDs: 1) use different brands. 2) use "too large ones" intentionally. 3) run a mirror vdev consisting of three devices instead of just two. 4) make backups to physically separate devices (a PBS) more often than usual.

PS: "mirrored raid-z" makes no sense. "Mirrored" describes multiple (usually identical) devices which contain the exact same content. "RaidZ = RaidZ1"/RaidZ2"/"RaidZ3" describes a construct which distributes the data plus one (two/three) sets of parity data onto several devices.

Thank you @UdoB , that what i meant using too large devices intentionally, more than the 2Gb/1Tb rule mentionned by @fabian somewhere on another discussion in this forum.
I don't want to use consumer grade ssd, i have to. The only ssd datacenter disks i already have seen on sale here are refurbished ones. The others brand new ones i have seen come as part of an extreemely expensive DELL-EMC storage and backup solution.
P.S: Thank you, I meant mirrored zfs volume.

 

[Johannes S]

Thank you @UdoB , that what i meant using too large devices intentionally, more than the 2Gb/1Tb rule mentionned by @fabian somewhere on another discussion in this forum.
I don't want to use consumer grade ssd, i have to. The only ssd datacenter disks i already have seen on sale here are refurbished ones.

Even a refurbished dc ssd will propably last longer than a consumer device of your boss won't veto it.

 

[IsThisThingOn]

Even a refurbished dc ssd will propably last longer than a consumer device of your boss won't veto it.

Not necessarily. Prosumer or NAS SSD have become pretty good.
For 100€, I can get a refurbished 800GB DC-S3510 with 450TBW or a brand new WD Red™ SA500 1TB with 600TBW and a 5y warranty.

 

[Johannes S]

Not necessarily. Prosumer or NAS SSD have become pretty good.
For 100€, I can get a refurbished 800GB DC-S3510 with 450TBW or a brand new WD Red™ SA500 1TB with 600TBW and a 5y warranty.

But they still lack power-loss-protection or did I miss something?

 

[IsThisThingOn]

But they still lack power-loss-protection or did I miss something?

PLP isn't a security feature, it only speeds up sync writes.

 

[Johannes S]

PLP isn't a security feature, it only speeds up sync writes.

True but since the topic of this thread is how to speed up PBS I think it's a good thing to have

 

[Fathi]

PLP isn't a security feature, it only speeds up sync writes.

I though DRAM cache speeds up writes and that plp, provided by capacitors, is a security feature.

 

[IsThisThingOn]

True but since the topic of this thread is how to speed up PBS I think it's a good thing to have

TBH, I don't know if PBS uses sync writes. I guess they are not needed and you could just send an ack when finished?
At least my backup to an NFS share seems to work that way. Fast transfer, reaching 100% and then there is still roughly 10s delay until it really finished.

Also OPs biggest issue is garbage collection, and I think this is a read only operation?

I though DRAM cache speeds up writes and that plp, provided by capacitors, is a security feature.

It is a "security feature" that you need because you did an insecure thing before.
It is like jumping out of a plane and calling the parachute a security feature.
Sure it is, but only because you jumped out of a plane. It is not a security feature when driving a car.

The SSD lied about data being written to NAND.
Because of that lie, the sync write is faster. But also because of that lie, data would be lost when there is a power outage.
So to make up for that, PLP drives have capacitors. Thanks to them, the drive can write stuff from cache to NAND even without the host having power.

There is one specific edge case you can rule out with PLP.

A word of caution: In the past, there were none PLP SSDs that lied about sync. A twitter user found out by running some test. Unfortunately the twitter user deleted his account, so I can't share it with you. Some Phison E18 controllers in combination with bad firmware from ADATA and Patriot lied about sync. That is why I don't blindly trust cheap consumer drives.

Source

I would still not call PLP a security feature because of that.
IMHO, this is more of a good example why you should get a mix of brands and controllers for a special vdev mirror.