ZFS raidz missing lots of disk space

[h00bi]

Hello,

i have a zfs pool with raidz(1) on 5x 960GB = 4,8TB Enterprise SSD. Should be roughly 4* 960GB = 3.84TB net capacity (showing 3,71TB)




There are less than 2,8TB VM disks, no CTs


blocksize is 8k, Thin provision is disabled.
EDIT: ashift=12

When i was trying to create a 2900GB drive instead of the 2500GB drive (2.68TB in picture3) i received out of space error messages.

Where is all the space going?
Should i change block size? Usage is mostly large files.

compression is off. Which numbers might go down if i enable lz4 compression and rewrite all files? does lz4 make sense without thin provisioning?

 

[aaron]

Have a look at this: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_zfs_raid_considerations

compression is off

This is a bad idea, as ZFS will also write out all zeroes.

from man zfsprops:

When any setting except off is selected, compression will explicitly check for blocks consisting of only zeroes (the NUL byte). When a zero-filled block is detected, it is stored as a hole and not compressed using the indicated compression algorithm.

 

[h00bi]

for most of the vm disks, 16K volblocksize is shown.


Datacenter -> Storage shows this:

 

[leesteken]

This surprises people often, but raidz padding make data take more space than expected: https://forum.proxmox.com/search/6582756/?q=raidz+padding&o=date

EDIT: Maybe this table can help: https://www.delphix.com/blog/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz

 

[h00bi]

So whats my actual zfs blocksize on this?