Clarification on "Block Size" 16k suggestion vs actual pool ashift 12

Mikesco3

Active Member
Sep 10, 2019
7
0
41
44
Hi Everyone...

I've already tried searching a bit on the forum and the Proxmox documentation.

I want to ask for some clarification about the "block size" (ashift) value suggested on the Proxmox Web Interface when adding or creating ZFS pools.

Lately, the UI has been suggesting a 16k block size, which would correspond to an ashift=14, (though I even remember for a while I was being suggested 8k instead).

However, when I check the actual pool after creation:
Code:
zpool get all rpool |grep ashift

# I get:
rpool  ashift                         12                             local

Usually I'm finding the pool using ashift=12 which corresponds to a 4k alignment.
So it looks like the UI recommendation and the actual ZFS ashift value are not always aligned.
I’m wondering if this difference is intentional, and whether the suggested “block size” in the UI is meant as a hint or as an actual configuration parameter.

My setup​

I typically install Proxmox on a pair of large SATA drives (mirrored), and then use a separate NVMe drive (or mirror) for production VMs.
I also replicate the VM virtual disks from the NVMe pool to “shadow” copies on the HDD pool, so that if an NVMe drive fails I can spin up or restore those VMs quickly.

So in effect, I have two pools:
  • One for VM production (NVMe)
  • One for backups / replicas (HDD mirror) in rpool

I’ve been using the default ashift=12 (4K) for both, assuming that would match most modern drives.
But since the UI now shows “16K,” I’m wondering:
  • Is that 16K "block size" supposed to match the ashift value of the underlying pool?
  • When there is a mismatch between the "block size" shown on the web interface and the actual ashift value of the pool, which one prevails? and what are the potential issues when they don't?
  • What is the relationship between the “block size” shown in the web interface and the actual ZFS ashift value used by the pool?
  • Should we be using an ashift value of 14 on the underlying NVME drives?
  • Is this documented somewhere?

I’d appreciate any clarification on what the Proxmox UI’s block size setting represents, and whether it’s something we should be matching manually to the pool’s ashift
 
  • Is that 16K "block size" supposed to match the ashift value of the underlying pool?
No.
  • When there is a mismatch between the "block size" shown on the web interface and the actual ashift value of the pool, which one prevails? and what are the potential issues when they don't?
They are different things and don't conflict.
  • What is the relationship between the “block size” shown in the web interface and the actual ZFS ashift value used by the pool?
The block size is the volblocksize and the ashift is the ashift.
  • Should we be using an ashift value of 14 on the underlying NVME drives?
No. It's unclear that the best ashift for your specific device is. Maybe test (real-world) performance? A erasure-page-size is probably way to big and 512 is rather small, so 12 is probably a good generic middle ground for ashift.
  • Is this documented somewhere?
There is lots of documentation about ZFS on the internet and ZFS, the ashift and the volblocksize are not at all Proxmox specific.

Larger volblocksize gives more data per IOP and more chance for compression to have an effect (as you cannot compress anything to less than an (2 to the power of) ashift block). ZFS documentation recommends 16k (double the 8k they recommended some years ago). The ashift of 4k is recommened for hardware that uses larger (than 512 bytes) sectors.

EDIT: For containers you want to think about different recordsize setting (instead of volblocksize).
 
Last edited:
  • Like
Reactions: Mikesco3
No.

They are different things and don't conflict.

The block size is the volblocksize and the ashift is the ashift.

No. It's unclear that the best ashift for your specific device is. Maybe test (real-world) performance? A erasure-page-size is probably way to big and 512 is rather small, so 12 is probably a good generic middle ground for ashift.

There is lots of documentation about ZFS on the internet and ZFS, the ashift and the volblocksize are not at all Proxmox specific.

Larger volblocksize gives more data per IOP and more chance for compression to have an effect (as you cannot compress anything to less than an (2 to the power of) ashift block). ZFS documentation recommends 16k (double the 8k they recommended some years ago). The ashift of 4k is recommened for hardware that uses larger (than 512 bytes) sectors.

EDIT: For containers you want to think about different recordsize setting (instead of volblocksize).
Thank you for the reply