ZFS and Failing to build an array on 24 drives

[gtaylorstc]

I am running a server with 24 - 2TB drives on an Areca 1280 Raid controller with all disks in JBOD mode. I have tried both RAIDZ2 and RAIDZ3 and added all 24 drives, assuming the hot spares are built into the array by Proxmox. Yet every attempt fails. I can build smaller arrays. It spits out this error when it fails.


command '/sbin/zpool create -o 'ashift=16' ZFS01 raidz2 /dev/disk/by-id/scsi-2001b4d204b20cab3 /dev/disk/by-id/scsi-2001b4d20bfde737a /dev/disk/by-id/scsi-2001b4d20230427a9 /dev/disk/by-id/scsi-2001b4d206991ec9f /dev/disk/by-id/scsi-2001b4d20aa23eeff /dev/disk/by-id/scsi-2001b4d20f8ae59df /dev/disk/by-id/scsi-2001b4d203c21f6d5 /dev/disk/by-id/scsi-2001b4d20d774b28e /dev/disk/by-id/scsi-2001b4d203bf9fe40 /dev/disk/by-id/scsi-2001b4d20ae2fda73 /dev/disk/by-id/scsi-2001b4d2002de4c1b /dev/disk/by-id/scsi-2001b4d20a7046a91 /dev/disk/by-id/scsi-2001b4d2089a2b24d /dev/disk/by-id/scsi-2001b4d2021f40e25 /dev/disk/by-id/scsi-2001b4d20652fa2bd /dev/disk/by-id/scsi-2001b4d208e3fee9d /dev/disk/by-id/scsi-2001b4d205764c273 /dev/disk/by-id/scsi-2001b4d20abdcb16a /dev/disk/by-id/scsi-2001b4d20dad870d0 /dev/disk/by-id/scsi-2001b4d20f1d86237 /dev/disk/by-id/scsi-2001b4d201ca1c21c /dev/disk/by-id/scsi-2001b4d206dbf734d /dev/disk/by-id/scsi-2001b4d2061d9f533' failed: exit code 1

 

[floh8]

Why ashift=16?

 

[guruevi]

Many reasons, if the disk already contains partitions/GPT data, or the disks are not all of the same size. Also, I would not recommend 24 disks in a RAIDZ2, the recommended is ~6-8 disks per RAIDZ2 and 8-12 in RAIDZ3 erring on the smaller sizes for data safety.

 

[MarkusKo]

i'm not a zfs expert but i read in some online posts that from an overhead pespective 6 or 10 drives ist the best option for zfs (Z1,Z2) pool.
with 24 drives you should create multiple pools or two mirrored pools. zfs draid would be an option too, it is intended for pools with 10+ drives.

as guruevi already mentioned you should check every drive if there is already a partition or someting else present, that would prevent creating a new array.

 

[ant12]

I use 24 drives in a concatenated 3X raidz2! 8 disks in every raidz2 and raid0 them!

 

[Kingneutron]

> assuming the hot spares are built into the array by Proxmox

To the best of my knowledge, you should not be using ashift=16. If these are spinning disks use ashift=12; if SSD you can try ashift=13

Also, NO HOTSPARES are defined in your command. Sysadmin creating the pool is the one specifying those at creation, or adding them in post-creation; proxmox has nothing to do with it.

You should also be subdividing those 24 drives into 3 vdevs of 8, or create the pool as a draid.

e.g.

zpool create -o autoexpand=on -o ashift=12 \
ZFS1 \
raidz2 /dev/disk/by-id/disk1 disk2 disk3 disk4 disk5 disk6 disk7 disk8 \
raidz2 9 10 11 12 13 14 15 16 \
raidz2 17 18 19 20 21 22 23 24 \
spare /dev/disk/by-id/blah1 blah2

If pool creation is failing, do a ' wipefs -a ' on /dev/sdX for all drives that will be part of the pool first. Then use commandline to create it, NOT the proxmox GUI.