Use ZFS Special vdev with Raid 5

[joshiyamamoto]

Hi all,

I'd like a sanity check on a plan before I execute it, and I'm aware it's not the textbook ZFS setup – so I'd appreciate honest opinions on the risk.

Problem:
Garbage collection on my main PBS datastore (storage1) takes ~28–46 hours per run. The datastore holds ~16 million chunks, ~17.5 TiB on-disk, dedup factor ~90. I've already maxed gc-cache-capacity (8388608), and the last GC log shows a 96% chunk-cache hit ratio, so the software side is exhausted. The bottleneck is clearly metadata I/O (atime updates in phase 1, chunk stats in phase 2) on slow spinning disks. GC is a pure metadata workload, so a ZFS special vdev on SSD is the obvious fix.

Hardware:
- Dell PowerEdge, PERC H740P (RAID mode, 8GB cache, BBU status: Ready), firmware 51.16.0-5150
- storage1: 4x HDD in hardware RAID5, presented as a single ~32TB virtual disk, currently XFS
- A second datastore (storage2) on its own separate hardware RAID5 on the same controller – I do NOT want to touch this one
- OS on a separate RAID1 (also on the PERC)
- 2x 240GB SATA SSDs (Micron, enterprise) on a separate Dell BOSS/AHCI controller, currently unused, exposed natively (real SMART/serials)

Why not the clean way (HBA + raidz1):
The H740P only supports non-RAID/pass-through in Enhanced HBA mode, which is a global controller mode change. That would tear down the storage2 RAID5 and the OS RAID1 as well, which I can't do, storage2 is ~38TB and would need to be fully evacuated too, and the OS would need reinstalling. So per-disk pass-through for just the storage1 disks is not possible on this controller.

My proposed plan (storage1 only):
1. Fully sync storage1 to a second PBS (separate hardware) and verify it
2. Keep the PERC in RAID mode. Leave storage2 and OS untouched.
3. Destroy the storage1 XFS, then build a ZFS pool where:
- the existing RAID5 virtual disk is the single data vdev
- the 2 BOSS SSDs are a mirrored special vdev (special mirror, ZFS-managed – not a controller RAID1)
4. Recreate the datastore, restore gc-cache-capacity, sync the data back.
5. Set special_small_blocks later to also pull the small index files onto the SSDs.

What I understand the trade-offs to be:
- No ZFS self-healing on the data side (ZFS sees one "disk"), but it still detects checksum errors, and I rely on hardware RAID5 redundancy + PBS verify
- Write-hole risk is mitigated by the healthy BBU (write-back is safe with a working BBU).
- The special vdev is structurally critical losing it loses the pool, so it's a mirror if one disk fails I can swap very fast that faulty disk to minimize risks

My question:
For a backup datastore, is ZFS on a single hardware RAID5 + mirrored SSD special vdev a reasonable way to get the GC speedup, or are there failure modes I'm underestimating?

Thanks a lot for any input.

 

[Johannes S]

AFIK special device is not supported in such a setup, @Falk R. mentioned this in the German forum.
Are you using your RAIDs battery cache?

RAID10 should help for performance too.

 

[daanw]

Why not keep storage2 and OS on the current PERC740P and get another HBA for creating a new zpool with direct access to the disks and special device?
If you have enough disks you can first do an offline copy of the current datastore from storage1 to the new zpool before removing storage1 from the PERC.

 

[waltar]

You would even have the following options:
1) Define hold metadata in fs cache and read once all files eg by find or ls -R into memory after a reboot,
2) create a xfs special device on ssd raid1 and a new xfs on the raid1+raid5 so a reboot don't hurt metadata access anymore,
3) define bigger dir's on xfs creation for far less I/O to read many files in dir's,
4) combine all together ...
but it's on your side

 

[IsThisThingOn]

PBS is mostly metadata read, so instead of svdev I would go with L2ARC.
There you are free to use one drive or three drive in RAID0 or whatever.

PS: please not that I would highly discourage you from such a wonky setup in general. RAID5 or RAIDZ1 is dead for HDDs.

 

[tcabernoch]

- the existing RAID5 virtual disk is the single data vdev

There's another option, you know. On the PERC, you make single-disk-raid0 for each drive you want to be member of the ZFS pool.
Folks will tell you that's horrible. But, they will tell you its horrible to even use HDD, throw up their hands and exclaim how awful you are because you don't have piles of SSD to work with.
I think that destroy the raid5 and make single-disk-raid0's out of the disks is a better option than building on top of a PERC raid, and given the various constraints you've listed, I'm guessing you don't have a lot of choices about this hardware.

If you do a special vdev, update us with your final special_small_block and recordsize numbers.
This is a hard problem. Getting those two numbers right is difficult.

You might want to read this.
https://www.seventhsignal.eu/blog/zfs-special-vdev-proxmox-backup/

 

[waltar]

RAID5 or RAIDZ1 is dead for HDDs.

That's right but only 4 hdd's are given. raid1+raid10 with hybrid-xfs works too as for a raid1+raid6 you want much more hdd's.
But anyway doing raid1/raidz/raid5 without configured hot-spare is quiet risky !!

 

[Johannes S]

PS: please not that I would highly discourage you from such a wonky setup in general. RAID5 or RAIDZ1 is dead for HDDs.

While I would agree with RAIDZ1 for obvious reasons, RAID5 can still be useful if (and only if!) the controller has a battery cache to speedup the HDD access. Although it would propably still perform better with a switch to RAID1/RAID10

 

[IsThisThingOn]

But anyway doing raid1/raidz/raid5 without configured hot-spare is quiet risky !!

It is even risky with a hot spare.

RAID5 can still be useful if (and only if!) the controller has a battery cache to speedup the HDD access.

Ignoring that battery cache only helps with sync writes compared to ZFS, it is not about performance but resilver times (safety).

But, they will tell you its horrible to even use HDD, throw up their hands and exclaim how awful you are because you don't have piles of SSD to work with.

HDDs are perfectly fine. Ignoring pool geometry and padding for PVE is not and why you should use mirrors instead.
PBS is thankfully a lot performance intensive. I went with SSDs for PBS, because of lower power consumption, and being not that much more expensive than HDDs. But I am pretty sure the only thing you need is metadata read performance.

Thankfully that is easy to test. Just add a L2ARC.

Special vdev on the other hand will not boost write performanc that much (because it is mostly bigger chunks written, not metadata). What I would bet does not work is boosting performance by changing the special_small_block value to something else than 0. 0 Will only store metadata, that one is huge. But setting it to something like, 512k or 2M will do almost nothing. Most of the time you won't read anything besides metadata. Or you do, if you restore, but then reading some 512k chunks from the SSD instead of HDD won't do much.

 

[bofh]

just add a new HBA, thats 200 bucks and it saves you a ton of headaches

 

[J-Rod]

What version of PBS? If you scan the PBS roadmap (https://pbs.proxmox.com/wiki/Roadmap) there have been improvements to garbage collection, particularly in 3.4 and 4.0.

Are you ensuring that garbage collection is not running while backups are running?

How much RAM is in the server? How much is typically used during garbage collection as a disk cache? More RAM might help if all available RAM is being used.

What hard drives are you using? There is a difference in performance between 5900RPM or 7200RPM SATA and 7200RPM Enterprise SAS.

Check the RAID controller settings for that array. Is write back enabled? Read ahead adaptive/enabled/disabled? I think disabled would be best in this use case but maybe run some tests.

Single parity RAID obviously has major shortcomings for data safety, but also any parity stripe based RAID has major performance implications for random I/O, and a RAID controller with 8GB can only mitigate so much. So switching away from this RAID5 setup might be the more important aspect to consider here.

Special device or no? I personally did not run any testing with my PBS system before/after, but I can tell from the I/O stats that the special device gets hammered with every PBS task, so my opinion is that it's a must-have. For sizing, I can only say that my special device usage is about 0.5% of the backup datastore usage (special_small_blocks=4K based on https://pbs.proxmox.com/docs/sysadmin.html#zfs-special-device).


As a wider conversation, I will recommend having multiple PBS servers, some flash, some HDD. Use an aggressive backup schedule to flash first, or if space constrained, at least the top high churn VMs to flash. For HDD, lower churn VMs, as well as long-term/archive backups via sync pull from the flash PBS using a weekly schedule to reduce I/O load on the HDDs (only pull the most recent weekly or daily backup, not all). I'd also highly recommend running all PBS instances as VMs, for maximum flexibility, portability, etc.