ZFS wrong space available

[Piuttosto Orbo]

Hello everyone
i've read some similar post, but i can't find a solution
I've tried with SSH and gui, but the error is the same in 2 different nodes with different disks and size
I have 11 disks 1TB (931,51 GiB) and i create raiz1 pool with LOG caches on SSD, but availabre space is only 8,78TB instead 10.

I have read someone who advice to change block size, but it set on 8k and it's impossible to change

Where is my error?

 

[apoc]

Dont get your issue.
11 disks minus 1 parity is 10.
10x 931 GiB = 9310 GiB
9310 GiB is roughly 9TiB which is pretty close to what you see... I wouldn't bother regarding these discrepancies
All the best

 

[Piuttosto Orbo]

Thanks tburger
but i can create only 6TB disk it's strange.
In another node i have 9x2TB in the same configuration and i can create only 8TB disk

 

[apoc]

Had the same issue in raidz2/3
This is related to the block size
You need to create the volumes with larger chunks.
Check this thread:
https://forum.proxmox.com/threads/u...reated-on-raidz3-pool-vs-mirrored-pool.65018/
All the best

 

[Piuttosto Orbo]

zpool is empty and i can destroy it.
How can i create a zpool with 64k blocksize? With this "zfs set recordsize=64K" ?

 

[apoc]

Have you read the referred thread?
Everything behind is explained there!
You just need to create the zvols different...

 

[Piuttosto Orbo]

i read it, sorry for my english, and i understand that you choose 64k volume blocksize, but i don't know how to create it.
i used this command
zpool create zfs-ml /dev/sdb /sdc [....] log /dev/sdn for the pool

then i add as stoage using 64k blocksize
but the problem still remains

zpool list show me correctly 10TB, proxmox show only 8,61, and i can't create disk larger than 8,5TB

i have to create a zvol? how to use zvol instead zpool as proxmox storage?
i tried
zfs create -V 64G -o compression=lz4 -o volblocksize=128k zfs/dati
but i can't see it

 

[apoc]

What do you mean with "I can't see it"?
Where?
Which output are you referring to?

 

[Dunuin]

I'm not sure if he is just confused because sometimes tebibytes and sometimes terrabytes are used and both are just presented as "T" not "TB or TiB".

looks like zpool is showing the size "10 T" which should be "10 TB" if he uses 11x 1TB drives in raidz1:
10 TB = 10 terrabyte = 10 * 1,000 * 1,000 * 1,000 * 1,000 = 10,000,000,000,000 bytes

proxmox gui is showing the size "8,61 TiB" in TiB so looks like the zfs command is showing the "8,61 T" also as TiB and not as TB like the zpool command:
8.61 TiB = 8.61 tebibyte = 8.61 * 1,024 * 1,024 * 1,024 * 1,024 = 9,466,795,115,151 bytes

You always loose some space and your free space on the zfs pool is 8.61 TiB = 9.47 TB whats not much less then the optimal raw capacity of 10 TB.

Thanks tburger
but i can create only 6TB disk it's strange.
In another node i have 9x2TB in the same configuration and i can create only 8TB disk

What do you mean with "create disk"? You mean creating a virtual hard disk for your VMs that can not be bigger then 6TB or 8TB? These would be called vdevs and are created atop of your zfs pool or datasets.

 

[apoc]

Don't know but to me it sounds more like the padding issue on RAIDZ which I have experienced as well...
Need to wait until we get some clarification

 

[LnxBil]

First, don't use such strange setups of 11 disks in one raidz1 pool. The speed is so slow ... you will not be able work with it properly. Use multiple vdevs for better speed (IOPS), but more waste of disk space in an optimal setup.

To compute the optimal way to store the data, you need to align everything correctly, so that you will not waste any space. This is a simplification (!) and not the ground truth, but just shows the order of magnitude of problem we need to tackle: the low level setup is the ashift value of your pool. So with 11 disks, you have a theoretical maximum of 11*512 or 11*4096 block sizes (without any redundancy) and that number is already very strange (not based on basis 2) and one vdev in your setup, so we can remove the vdev from the equation and just assume a 10 disk setup with 10*512 or 10*4096 disks. If you now write a volblock of e.g. 8K, this block is divided by 10 and stored on a block on each disk that has an ashift of 9 (512), we need to write two blocks to each disk of a total of 820 bytes, so we "waste" 204 bytes (20%). In the ashift 12 (4096) case, we waste even more 4096-820 (80%). The higher the volblocksize (as @tburger already pointed out), the waste percentage lowers, but can never by 0 due to the strange setup of 10 (or 11) disks. If you would use a number of disks that resemble more of a base-2 exponent, you can lower the waste in a raidz1 system.

This is also true for mirrored disks, but the are intrinsically already based on base-2 exponents, so the waste is already minimal in such a setup.