zfs TB eater

[ledufinfra]

hello,

i'm ... really disappointed.
i have 4TB sas drive ... so decided to build ZFS RAIDZ (that allow me to lost one drive : good ) ... so from 16TB i'm going to 12TB usable , ok
But NO !
i want to add 9000 GB (9TB) HDD for my openmediavault VM .... and this system said to me : blabla , you run out off space (error 500), what ??? out of space with 12TB ?

 

[LnxBil]

Let me search that for you:

• https://forum.proxmox.com/threads/raidz-out-of-space.112822/
• https://forum.proxmox.com/threads/cannot-store-on-zfs-raidz-volume-out-of-space.132035/
• https://forum.proxmox.com/threads/where-is-my-zpool-storage.123937/#post-539665
• https://forum.proxmox.com/threads/where-is-my-1-2tb-goes.124046/

 

[ledufinfra]

Let me search that for you:

• https://forum.proxmox.com/threads/raidz-out-of-space.112822/
• https://forum.proxmox.com/threads/cannot-store-on-zfs-raidz-volume-out-of-space.132035/
• https://forum.proxmox.com/threads/where-is-my-zpool-storage.123937/#post-539665
• https://forum.proxmox.com/threads/where-is-my-1-2tb-goes.124046/

thank you. weird thing : 16TB , then 12TB ... then 8TB usable ???
this is just a TB waste : loosing 50% ????
ZFS is definitively not a good choice , if you want to be power friendly.

 

[showiproute]

@LnxBil Thanks for the advices - I was also not aware of that.
Does this also apply to pure storage in example when using a RaidZ for PBS?

 

[Dunuin]

ZFS is definitively not a good choice , if you want to be power friendly.

Don't blame ZFS. You lose that additional 4 TB because you don't set that pool up well. Thats just a user error. Read about padding overhead and the volblocksize and you could use your 12TB. But I still wouldn't use the full 12TB as ZFS always needs some free space for proper operation. I wouldn't recommend filling it more than 90%.

@LnxBil Thanks for the advices - I was also not aware of that.
Does this also apply to pure storage in example when using a RaidZ for PBS?

Padding overhead only affects zvols but no datasets. As PBS is using datasets this shouldn't be a problem.

 

[ledufinfra]

Don't blame ZFS. You lose that additional 4 TB because you don't set that pool up well. Thats just a user error. Read about padding overhead and the volblocksize and you could use your 12TB. But I still wouldn't use the full 12TB as ZFS always needs some free space for proper operation. I wouldn't recommend filling it more than 90%.


Padding overhead only affects zvols but no datasets. As PBS is using datasets this shouldn't be a problem.

ok, so what you recommend me to do ? (sorry i'm not a zfs expert as u can see :-( )
redo ZFS on my 4 x 4Tb disk (3.6TB) in RAIDZ1 but which option must i choose ?

 

[showiproute]

May I also add a point here found there:

• Sequential Workloads: If the I/O pattern is mostly sequential, such as in large file transfers, backups, or streaming, a larger block size (like 128K) can be efficient. Larger blocks mean fewer I/O operations for the same amount of data, making them more suited for sequential tasks.
• Random Workloads: If the I/O operations are mostly random, as seen in databases or virtualization environments, a smaller block size (like 8K or 16K) might be more appropriate. Smaller blocks reduce the risk of I/O amplification where reading or writing a small amount of data requires the system to handle a much larger block.

What would be the appropriate way to handle that: In general the underlying VM is a mostly a file server with sequential writes - should we choose a higher blocksize for the specific disk?
Or does ZFS on Proxmox handles this different and a smaller blocksize should be choosen?

 

[ledufinfra]

i put block size of 32K ... and i can allocate 8192GB for my nas ... but at the end this is always 4x4TB (16 TB) divided by two !!!
ashift stay at 12 (and i do not understand the meaning of that)

 

[Dunuin]

i put block size of 32K ... and i can allocate 8192GB for my nas ... but at the end this is always 4x4TB (16 TB) divided by two !!!
ashift stay at 12 (and i do not understand the meaning of that)

With a default blocksize of 8K you will lose 50%. With 16K you will lose 33%. With 64K you will lose 27%.

And the blocksize is defined when creating the virtual disk. So you will have to destroy and recreate all virtual disks (doing a backup+restore would be the easiest way) after changing the block size of your ZFS storage.

You can check the blocksize of your zvols with zfs get volblocksize.

 

[showiproute]

With a default blocksize of 8K you will lose 50%. With 16K you will lose 33%. With 64K you will lose 27%.

And the blocksize is defined when creating the virtual disk. So you will have to destroy and recreate all virtual disks (doing a backup+restore would be the easiest way) after changing the block size of your ZFS storage.

You can check the blocksize of your zvols with zfs get volblocksize.

But will that also affect the performance of the underlying VM as mentioned in my previous post?

 

[Dunuin]

But will that also affect the performance of the underlying VM as mentioned in my previous post?

Yes. Especially read/write amplification when doing reads/writes smaller than your volblocksize. So running volblocksize=32K would be terrible when
running any MySQL or PostgreSQL DBs doing 8K/16K IO. Then doing a 4K write would mean read 32K + write 32K.
Compression ratio also depends on the volblocksize as bigger blocks are better compressible. On the other hand a smaller volblocksize would allow for more efficient native deduplication.
And a bigger volblocksize means a better data to metadata ratio, so in theory faster when doing a lot of big sequential IO.

 

[leesteken]

But will that also affect the performance of the underlying VM as mentioned in my previous post?

If you want better write speed (and are OK with 50% space, like your original situation), use a raid10 (stripe of mirrors) with those 4 drives and you can chose any volblocksize.

 

[showiproute]

Yes. Especially read/write amplification when doing reads/writes smaller than your volblocksize. So running volblocksize=32K would be terrible when
running any MySQL or PostgreSQL DBs doing 8K/16K IO. Then doing a 4K write would mean read 32K + write 32K.
Compression ratio also depends on the volblocksize as bigger blocks are better compressible. On the other hand a smaller volblocksize would allow for more efficient native deduplication.
And a bigger volblocksize means a better data to metadata ratio, so in theory faster when doing a lot of big sequential IO.

I use this zfs pool mostly for storing films and audio stuff.

Will that also be a benefit on a single vdev/zpool or only for raidz?

 

[Dunuin]

Will that also be a benefit on a single vdev/zpool or only for raidz?

In general, not limited to raidz. But would be probably annoying to change asyou would need to destory and restore all your Films for that...

 

[showiproute]

In general, not limited to raidz. But would be probably annoying to change asyou would need to destory and restore all your Films for that...

Just moving the VM disk from one storage pool to another and back again?

 

[Dunuin]

Just moving the VM disk from one storage pool to another and back again?

Yes, but you need enough space for that vdisk on that other storage.