ZFS config: which RAID level for 12x 20TB SAS hdds?

[Tony]

Hi,

I am in the progress of setting up a Proxmox VE server with ZFS, using 12x 20TB SAS disks in HBA mode. Which ZFS RAID level would be a good choice? I am thinking of the following options:

(1) RAID10: 6x mirror vdev
(2) RAIDZ2-0: 2x RAIDZ2 vdev, each 6 disks, then RAID0 from those 2x RAIZD2 vdev

(1) seems to be recommended for large disks; however (2) would give a bit more tolerance and usable space in trade off for performance and re-silvering time. We don't require top performance from this pool.

Thanks in advance for any hint.

 

[aaron]

and usable space

Do you want to store VMs? Then you will most likely be in for a surprise... Check out the following section in the docs if you haven't seen it yet https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_raid_considerations

 

[ness1602]

You could always use raidz3 and a spare.

 

[Tony]

Do you want to store VMs? Then you will most likely be in for a surprise... Check out the following section in the docs if you haven't seen it yet https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_raid_considerations

thank you very much for the hint. It looks like using raidz for VM disks is a bad idea, regardless number of disks and raidz level of the vdev.

Perhaps I should have 2 pools: 1x of raid10 (4 disks) and 1x of raidz2 (8 disks)? The idea is: vm on raid10 pool, static data + backup on raidz2 pool.

 

[LnxBil]

Don't ever use such large disks for VMs. If one disks fails, you will have a LOT of time until the resilvering is done and it'll be very, very slow in the meanwhile. The throughput of spinning disks scales absolute not, so while the disk size is increasing over the last years, the speed is not. For backups it's ok, but it'll also take very long. We use "only" 8 TB disks for external backups, which is already a real pain (even via SATA).

I you've already bought the disks, consider buying also two (or more) enterprise SSDs for using as metadata storage and space for some high speed application. The spinning rust part of your pool can then be used for sequential stuff (random will still be very slow).