Zpool config considerations

[Proxtof]

I'm quite new to ZFS and Proxmox, so I'm not sure what the best options are.
We are setting up a Proxmox Backup Server and are now thinking about how to setup the zpool and datastore. We have three SSD's of 1TB and 10 HDD of 24TB.
First plan was to set up three three-way mirrors and a spare disk (resulting in 72TB usable space), plus a special device with a mirror of the three 1TB's.
(it can handle at least two disk failure, up to 6 if all the "right disks" fail. if three disks in the same mirror fail, all will be lost)

Another option would be a RAID10 with 5 2-way mirrors. This would give about 120 TB, can handle 1 to 5 disk failures. 2 disks that fail might be problematic and does not have a spare

I was also thinking about RAIDZ config with two or three parity disks (+spare?)
e.g. 6disks + 3parity + spare, would give about 144TB I think. 3 disks can fail, and a spare is available. This setup is probably slower (parity calculations etc.), but probably enough for backup purposes. The special device would then be less than 1% however.


Any thoughts or recommendation?

 

[MarkusKo]

I'm not an expert on this but you should go for zfs equivalent RAID10 with the ssd special device for performance reasons. From what i've read on this forum especially restores seem to be very slow if you don't use ssd's or at least have a ssd backed special device for your zfs hdd pool.

Backup storage:
    Prefer fast storage that delivers high IOPS for random IO workloads; use only enterprise SSDs for best results.
    If HDDs are used: Using a metadata cache is highly recommended, for example, add a ZFS special device mirror.

https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements

 

[UdoB]

I'm not sure what the best options are.

Yeah, you're right, there is no easy "always right" construct. The following is my personal idea, not the answer.

You have already three devices for the triple-mirrored vdev. That's good! And it shows me that reliability is considered important here.

Usually striped mirrors ("Raid10") is the recommended setup for the rotating rust part, mainly for IOPS. If the risk of losing a complete vdev (and all data) is considered too high I would probably opt for two RaidZ2 vdev with 5 disks each. This way the redundancy level of all vdevs (two data plus that single SD) are identical.

This way any two devices may fail.

Just to be sure: those SSDs should be "Enterprise Class" w/ PLP.

Good luck.

Possible limitation: I am NOT sure if the effective IOPS of two spindles plus the SD for metadata will work on the long run, as all of my own similarly crafted PBS' are way smaller. If you go this way it is highly recommended to not only monitor the whole thing continuously but also to run an actual test-restore of a large(!) VM each month or so.

Just my 2 €¢...

 

[Johannes S]

Adding to what everybody already said I would also like to point out that as long as you still in the planning phase it might be worth to do intensive benchmarking and compare different results.

Personally I would go with the striped mirrors for the HDD and three-way mirror for the SSDs since this seems to be the best compromise between capacity, IOPS and redundancy.