Setting ashift for ZFS (and tuning in general)

[darin755]

So I have a few systems that are built out of cheap hardware. I am looking to try and get as much performance out of them as I can. Right now I am getting around 100mb/s write and 25mb/s read. Each system has different SSDs but fdisk shows 512 byte sector size for all.

Disk /dev/sda: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: PNY CS900 500GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

My ashift size is currently 12. Should I switch it to 9?

 

[waltar]

You could try ashift=9 but you will not be able to measure a difference and you are on "the better future side" if doing "=12".
When you have nvme with different LBA format support it depends on nvme type but mostly get better results with 4k LBA format when use zfs onto (while with xfs curiously LBA size 512 was better).

 

[darin755]

You could try ashift=9 but you will not be able to measure a difference and you are on "the better future side" if doing "=12".
When you have nvme with different LBA format support it depends on nvme type but mostly get better results with 4k LBA format when use zfs onto (while with xfs curiously LBA size 512 was better).

Do you think it is worth trying? The reason I am asking is that I noticed that a scrub runs at around 250mb/s. It is weird to me that my reads are so slow and they should be pretty fast especially when I only read a medium file that can fit entirely in ARC.

How would I find the bottleneck?

 

[waltar]

The crux at yours is that reads are so much slower as writes which is normally the opposite with ssd's. Funny problem ... don't know further yet

 

[Kingneutron]

> My ashift size is currently 12. Should I switch it to 9?

Yes, rebuild the pool with ashift=9 for better performance.

FYI if future disks have sector size above 512, you will not be able to do a straight ' zpool replace ' and will have to copy the data over to the ashift=12 pool

 

[darin755]

> My ashift size is currently 12. Should I switch it to 9?

Yes, rebuild the pool with ashift=9 for better performance.

FYI if future disks have sector size above 512, you will not be able to do a straight ' zpool replace ' and will have to copy the data over to the ashift=12 pool

Never mind. My test numbers were wrong (I retested with fio and it was much better around 100m/s)

The slow down I did notice was actually caused by one of my drives being shoved in a location with not enough airflow. The drive doesn't have a temperature value in smart but when I shutdown the system and felt the temperature is was very hot. I moved it to an area with air movement and the issue is resolved. If someone is reading this in the future make sure to validate your numbers and then if your numbers are still low check for silent thermal throttling.

 

[darin755]

The crux at yours is that reads are so much slower as writes which is normally the opposite with ssd's. Funny problem ... don't know further yet

It turns out dd doesn't give actuate results. I restested the slow system and got 70/70. When I corrected a thermal issue I got a little over 100m/s both read and write,