Best ZFS Configuration for 12 18 TB HDD Drives?

[LunarMagic]

I am thinking about what zfs configuration to use. I think RAIDz2 would be great because its 2 drives for tolerance, but some people were saying not to use it with more than 10 drives, do you think 12 drives would still be fine?

Thank you!

 

[Kingneutron]

RAIDZ2 should still be OK with 12 drives, but test how long a drive replace/resilver will take with the pool at least 50% full. Keep in mind, it will take longer than that if the pool is above 80% used.

If resilvering takes longer than 24 hours, you may want to consider breaking the pool into 2 vdevs (6x18TB RAIDZ2 + 6x18TB RAIDZ2). You will lose 2 more disks of free space, but rebuilds, scrubs, and general I/O should be faster.

You could also consider going with more, but smaller-sized, drives if you have the slots for it.

Remember to schedule at least monthly recurring scrubs, and have some kind of backup plan. You should also keep at least 1-2 spare drives around in case a drive dies, 18TB+ is a lot of data at risk when you have to replace a drive and wait for a replacement to ship.

/ personally I went with a 14-disk DRAID with 2-disk tolerance and 1 vspare on a 15-bay 3.5-inch disk shelf, but that's with 4TB SAS drives.

 

[LunarMagic]

Thank you so much for your help!!!

 

[Dunuin]

Jep, usually you would split that into smaller vdevs. Keep in mind that HDDs aren't getting faster for decades. A 18TB HDD is as slow as a 600GB one. But as it is 30 times bigger it needs 30 times longer when rebuilding the raid array / resilvering the pool.

 

[Ramalama]

i Have in one Server an Raid z2 with 8x 20TB, without any issues, its fast enough for a Movie Collection xD

12 Drives is even Cooler, Z2 is really not an issue, but you can even do 4x Z1 in a stripe, which brings more performance.
Z1 on ZFS is not an issue, since there is no write hole (if the server crashes during rebuilding).
I think it was once an issue long time ago, thats why people still think it is one, because zfs is simply so old. But there is none.

However do what you like, i don't think that you need to think about performance anyway, since with 12 Drives, almost in any combination, a 10GBe connection would be anyway the limiting factor.

  pool: HDD_Z2
 state: ONLINE
  scan: resilvered 23.3G in 00:02:07 with 0 errors on Sun Mar 10 04:44:03 2024
config:

    NAME                                                    STATE     READ WRITE CKSUM
    HDD_Z2                                                  ONLINE       0     0     0
      raidz2-0                                              ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7XXXX                    ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7XXXX                    ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7XXXX                    ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7XXXX                    ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG6XXXX                    ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7XXXX                    ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7XXXX                    ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7XXXX                    ONLINE       0     0     0
    special
      mirror-1                                              ONLINE       0     0     0
        nvme-eui.0025384831b022f3-part5                     ONLINE       0     0     0
        nvme-eui.0025384431414875-part5                     ONLINE       0     0     0
      mirror-2                                              ONLINE       0     0     0
        nvme-eui.00253844314008c6-part5                     ONLINE       0     0     0
        nvme-Samsung_SSD_990_PRO_2TB_S7DNNJ0WA01XXXX-part5  ONLINE       0     0     0

The Special VDEV for metadata, are some partitions from 990 Pro's that i use in another zfs pool with mdadm, is a bit complicated, but i have only 4x 990 Pros which i use for a fast VM-Storage pool, and i wanted additionally a special vdev on the HDD Pool.
The special vdev speeds up insanely the Windows-Searching via Samba on Windows etc...
Without a special vdev and that billions of Data on the HDD pool, the windows search would be unusable.

Additional note, stay away from 990 Pro's, these drives are the worst crap ever, they die tooooo fast
Im Replacing like every month or every second month an 990 Pro xD
980 Pros or PM9A3 are a much better option.

Cheers

 

[LunarMagic]

Thank you