ZFS ashift and SAS 512e vs 4Kn

[sking1984]

Hi all,

Wondering what everyone would do? Would you use ashift 9 with a 512e sector size disk or go ashift 12? My understanding of zfs is still fairly basic but it is growing and im not sure what way I should go. Theoretically the disk is making it capable of doing 512k sector size but will I get better performance with ashift 12?

Thanks

note: disks below
https://www.disctech.com/Toshiba-AL14SEB060N-600GB-Enterprise-SAS-Hard-Drives
https://www.disctech.com/Seagate-ST600MM0018-600GB-SAS-Hard-Drive (although this shows up as a X422 STBTE600A10 by NetApp in the system details, weird)

 

[mira]

Usually you want to use ashift 12 nowadays. Especially if those disks are actually native 4k with emulated 512.
Do you plan on using the ZFS Pool mainly for VMs or CTs?

 

[sking1984]

VM's and CT's, does that change things?

 

[mira]

Since VMs by default use a volblocksize of 8K, all reads and writes will be 8K, so you don't lose anything by going with ashift 12.
For CTs, since they use ZFS as filesystem, there may be a little overhead in theory, but if the disks themselves are 4Kn, it wouldn't help in any way to go with ashift 9.

 

[Dunuin]

Since VMs by default use a volblocksize of 8K, all reads and writes will be 8K, so you don't lose anything by going with ashift 12.

Block level compression will be worse and VMs on raidz1/2/3 will waste more space because of padding overhead when using ashift=12 and 8K volblocksize. For both you want the volblocksize to be a multiple of the sectorsize/ashift. So there a smaller ashift would help, as the vollbocksize then doesn't have to be increased that much to reach the same efficencies.

But ashift=9 doesn't make any sense except you got 512n HDDs and don't plan to ever replace a failed disk with a 4K one (or extend it with 4K disks).

 

[chrcoluk]

Hi all,

Wondering what everyone would do? Would you use ashift 9 with a 512e sector size disk or go ashift 12? My understanding of zfs is still fairly basic but it is growing and im not sure what way I should go. Theoretically the disk is making it capable of doing 512k sector size but will I get better performance with ashift 12?

Thanks

note: disks below
https://www.disctech.com/Toshiba-AL14SEB060N-600GB-Enterprise-SAS-Hard-Drives
https://www.disctech.com/Seagate-ST600MM0018-600GB-SAS-Hard-Drive (although this shows up as a X422 STBTE600A10 by NetApp in the system details, weird)

Better performance, and if its an SSD also better endurance, definitely go ashift 12.

Also if you ever replace a disk in pool later with a drive that doesnt have 512e support, this would prevent an even bigger problem, think ahead.

If worried about compression can increase volblocksize, I have found bigger blocks tend to help performance anyway on large writes and can be metadata savings.

 

[Dunuin]

If worried about compression can increase volblocksize, I have found bigger blocks tend to help performance anyway on large writes and can be metadata savings.

Jup, but for example it would be a bad idea if you plan to run any DBs. I wouldn't go over a 8K volblocksize for a postgresql DB (uses 8K sync writes) or over 16K volblocksize for a MySQL DB (uses 16k sync writes). Otherwise you will see very bad write and read amplification with a lot of SSD wear and lost performance.

 

[chrcoluk]

Yep there is always exceptions that need special attention.

 

[sking1984]

Just following up on this. Want to make sure I did the right thing. I used ashift=12 as seen below:

root@pve:~# zpool get all | grep ashift
pve1-data  ashift                         12                             local
rpool      ashift                         12                             local

draid- 3disk vdev's, 4vdev's, 14 disks, 2 hotspares, but a mix and match of disk manufacturers



First 2 disks are zmirror (HOST OS)
Second pool is 14 disks ashift =12 (HOST VM and LXC storage)