zfs topology and speeds

Apr 27, 2024
490
182
43
Portland, OR
www.gnetsys.net
Hello folks.

I've been looking at some of my zfs disk builds. Trying to optimize for speed.

This is an array of 8 identical spinners on a Gen13 Delll poweredge.

zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
-------------------------------- ----- ----- ----- ----- ----- -----
rpool 2.26G 4.36T 71 34 989K 351K
raidz1-0 2.26G 4.36T 71 34 989K 351K
scsi-350000397480bd199-part3 - - 18 8 261K 88.9K
scsi-350000397480bd259-part3 - - 17 8 238K 87.5K
scsi-350000397480b8b79-part3 - - 17 8 251K 88.3K
scsi-350000397480b8c01-part3 - - 17 8 239K 86.5K

So that's read 989K and write 351K.

And then I add another vdev and the read speed goes down?

zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
-------------------------------- ----- ----- ----- ----- ----- -----
rpool 46.4G 8.67T 0 57 3.69K 508K
raidz1-0 23.4G 4.34T 0 28 1.96K 253K
scsi-350000397480bd199-part3 - - 0 7 518 64.9K
scsi-350000397480bd259-part3 - - 0 6 462 60.3K
scsi-350000397480b8b79-part3 - - 0 7 549 64.9K
scsi-350000397480b8c01-part3 - - 0 7 479 62.7K
raidz1-1 22.9G 4.34T 0 28 1.73K 255K
scsi-350000397480ba0f9-part3 - - 0 7 476 65.5K
scsi-350000397480b6271-part3 - - 0 6 413 61.1K
scsi-350000397480bd34d-part3 - - 0 7 471 65.4K
scsi-35000039aa818b5c9-part3 - - 0 7 406 62.8K

read 3.69K and write 508K

All of the disks are reporting lower speeds.
Total combined write goes up, as you'd think it should, but the disks are slower.
Read speeds dropped to the byte range.
Those numbers aren't just bad, they are awful.

I have the host set in HBA mode.
Is there anything else here that I may have overlooked?
Why on earth would it do this?
 
Last edited:
Try zfs 3x mirror with ssd as zfs spezial device.
After Setup all your meta data will be located on this devices.
If you want this for old Starter you must write it again.
But only use Enterprise PNP SSDs. You can calculate approx. 2 % of your raw hdd space as ZFS special device size.
You can setup on your ZFS Pool atime as off.
 
Last edited:
  • Like
Reactions: UdoB