Best practice for PBS installation

[Vandros]

Good day.
For now I have, already installed and working, PBS instance on this hardware:
AMD Ryzen 5 Pro 3600, 32 GB ECC 2933 MHz, 2×512 GB SSD NVMe, 6×6 TB HDD SATA
The system is on 2x512 ssd with zfs mirror. Backup datastores are on 6x6 hdd with raidz2(about 21,6 tb of usable space). I have 7 PVE servers (not clustered) that making their backups on 7 datastores. The amount of data on these servers are from 1tb to almost 5tb (actual used space on PBS for all backup data with 5 days retention is about 9tb). The actual speed of backup is acceptable, but GC and verification processes are very slow. To somehow force the speed of jobs I set up GC job to run once a week. But following some recommendations i found here verification job must ran at least once a day. So they are continually starting after each other completion in forever loop.
I have another server in my arsenal: AMD Ryzen 7 Pro 3700, 32 GB ECC 2933 MHz, 2×480 GB SSD SATA, 4×14 TB HDD SAS. I think to install another instance of PBS on this server with proper configuration of disk subsystem. Can you give some suggestions on how to correctly install PBS on servers with HDD? I know about zfs cache and log but not sure how to properly config them.

 

[Dunuin]

PBS wasn't designed with HDDs in mind and a GC task needs to read the atime of every chunk. As a chunk can't be bigger than 4MB you should got atleast 2.25 million chunks (probably some million chunks more because of compression and so on). So each time you do a GC, you bascially force your HDDs to read millions of small files that are fragmented accross all HDDs. HDDs got terrible seek times and IOPS so that this takes so long.

If you just want to speed up the prune/GC task performance you could add a single SSD as a L2ARC and set it to "secondarycache=metadata", so that these cached atimes can be read from that SSD.

If you want to speed up the read/write IOPS of your HDDs in general, you could add 3 SSDs as a "special device" three-way mirror (you need 3 SSDs because it should match the reliablility of your HDDs, because if you loose the "special devices" all data on the HDDs is lost too).
That way the 6x HDD raidz2 would just store the data and the 3x SSD three-way "special device" mirror would just store the metadata.
Prune and GC tasks are very fast that way because all needed for that is metadata can can be read from the SSDs. And because the HDDs then no longer need to also store the metadata, the HDDs are hit by way less IOPS so write (even async writes!) and read performance will be way better for all random or small reads/writes. So this might even speed up backup/restore times and verify tasks a bit.

But only way to get that verify task really fast would still be to scrap that HDDs and to replace them with SSDs.

 

[Vandros]

Well that's pretty disappointing. I don't really have any other choices. These servers are from OVH, they don't have appropriate alternatives with ssd for fair price.
About these "special devices" - are they need to be the same capacity as HDDs? Also, do I need 2 SSD special devices for 4 HDD setup?

 

[mvs]

Daily verification is an abuse for HDDs. Modern high capacity drives have transfer rating of just around 500~750 TB/year.
Even with weekly verification you will probably exceed drive warranty limits.

To speed up GC/prune you can technically:
- use a mirror of only 2 SSDs as special device for RAIDZ or RAIDZ2 by overriding restriction with -f flag
- split SSDs in 2 partitions and use them as system drive and special device (i.e. 80GB OS / 400GB special)
- increase recordsize option

 

[Dunuin]

Well that's pretty disappointing. I don't really have any other choices. These servers are from OVH, they don't have appropriate alternatives with ssd for fair price.
About these "special devices" - are they need to be the same capacity as HDDs?

You can calculate the size. See for example here: https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954
Rule of thumb would be "0.3% of the pool size for a typical workload". So with 6x6 TB as raidz2 you would roughly need three 72GB SSDs.

Also, do I need 2 SSD special devices for 4 HDD setup?

"special devices" don't support raidz1/2/3 and should match the reliability of your data bearing disks. So ir depends on how you use those 4 HDDs. 2 SSD in a 4 disk raidz1 would be fine, 2 SSDs in a 4 disk raidz2 not.

 

[Vandros]

I can't find any information about special devices with raid10 zfs. Is it even possible? And how many disks will I need in special device with 4 hdds raid10?

 

[mvs]

I can't find any information about special devices with raid10 zfs. Is it even possible?

Why should it be not possible? You can store metadata on separate drive with any data vdev configuration.

And how many disks will I need in special device with 4 hdds raid10?

There is a concept of equal reliability, where a special vdev need's to have same redundancy level as data vdevs.
RAID10 has mirrored vdevs, so you need two drives combined in a mirror for a special vdev to match.

 

[mvs]

"special devices" don't support raidz1/2/3 and should match the reliability of your data bearing disks. So ir depends on how you use those 4 HDDs. 2 SSD in a 4 disk raidz1 would be fine, 2 SSDs in a 4 disk raidz2 not.

Yeah, this is the theory. But HDDs and SSDs have different reliability and as a consequence RAIDZ2 HDD pool is not necessarily more reliable then a mirror of two datacenter grade SSDs.