ZFS ashift on NVME SSD

[mindio]

Hi All,

What should be correct ashift setting (9 or 12) on zfs pool with such disks:

Disk /dev/nvme1n1: 1.46 TiB, 1600319913984 bytes, 3125624832 sectors
Disk model: Micron_7400_MTFDKBG1T9TDZ               
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

As far as I understand, physically it's 4k, but namespace is organised in 512b?

Thanks.

 

[Kingneutron]

For 4k sector, use ashift=12

 

[ucholak]

and use 4kn, format nvme with it

# nvme0n1
nvme id-ns /dev/disk/by-id/nvme-.. -H |grep LBA
nvme format /dev/disk/by-id/nvme-.. -l 3

 

[mindio]

I understand what ideal situation should be, but since I can't rebuild the pool right now, I want to understand how bad/inefficient current config is in terms of write amplification or else. I currently have ashift 12 on the mirror of these drives.

Also what's the reason behind enterprise grade drives comes preconfigured with LBA 512, while having 4k physical sectors?

 

[ucholak]

• 512e = inertia, compatibility
• from hw perspective it says like 512e = "good", 4kn = "best"
• if you have redundancy, or spare, you can rebuild raid, remove device, format, and re-add
• from benchmarks on internet it looks like 2-5% agregated diff, in special cases 2x, but it will be very hw related.
• i think it is worth it

 

[ucholak]

on vm level (config), not for existing vms, or migration required

# 4kn 
echo 'args: -global virtio-blk-device.physical_block_size=4096 -global virtio-blk-device.logical_block_size=4096' >> /etc/pve/qemu-server/100.conf

 

[LnxBil]

on vm level (config), not for existing vms, or migration required

# 4kn
echo 'args: -global virtio-blk-device.physical_block_size=4096 -global virtio-blk-device.logical_block_size=4096' >> /etc/pve/qemu-server/100.conf

Do you experience a difference in performance with this?

 

[MihSav]

https://www.techpowerup.com/ssd-specs/micron-7450-pro-15-tb.d1869
This page about 15Tb model, but very possible flash chips are the same. And their page size is 16K

 

[leesteken]

https://www.techpowerup.com/ssd-specs/micron-7450-pro-15-tb.d1869
This page about 15Tb model, but very possible flash chips are the same. And their page size is 16K

Interesting! The default ZFS volblocksize is currently also 16K (not sure if Proxmox also changed their default), so I already tend to use args: -global scsi-hd.physical_block_size=16k -global scsi-hd.logical_block_size=4k (but do that before installing any operating system inside the VM).

 

[ucholak]

i have setup 4kn on all levels from beginning.

for me, difference between emulated no-optimal setup, does not matter so much, because native should not be worse, and zfs+virtio cuts down iops+transfer performance to ~1/6 of native.

but should be interesting (and probably very hw depended).

Do you experience a difference in performance with this?

 

[ucholak]

https://www.techpowerup.com/ssd-specs/micron-7450-pro-15-tb.d1869
This page about 15Tb model, but very possible flash chips are the same. And their page size is 16K

my ~4tb 7450 pro's and max's only supports 512 or 4k.

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best (in use)

also other drives that i have seen (kioxia, micron 9300...). but for bigger sizes, bigger sectors makes sense.
specially, i have seen 8k on consumer Samsung drives somewhere.