ZFS for VMs - where did my hdd space go?

[sonnyj]

I’m totally new to ZFS and I need help with a ‘where did my hdd space go?’ question.

I’ve set up a RAIDZ1 ZFS pool (named Tank1) of 4 x 4TB drives (LSI Controller already in IT mode - I successfully tested passthrough to OMV but I'd rather avoid an extra layer):

Having read the forum, I concluded I need to set up zfs datasets to act as a storage for my virtual machines (please correct me if I'm wrong here).

After the zfs pool creation the ‘zfs list -t all -r -o space’ command returns available space as 10.4T:

​

I’m ok with that as I understand there is a need for metadata, etc. (but if I’m wrong I’m happy to recover that missing 1.5 TB of space)

I then created a dataset using the ‘zfs create -V 5T Tank1/surv’ command.

The ‘zfs list -t all -r -o space’ command returns:



The QUESTION: Is it normal for Proxmox zfs 5TB dataset to use 7.4TB zfs pool? (both gui VM add hdd and the cli as above yield the same result).

The main objective is to use those 4x4Tb disks as a ZFS storage for my survaillence camera footage and plex storage (separate container). I'd rather not use OMV (and similar) and just use basic proxmox mechanisms to avoid an unnecessary overhead. What am I doing wrong?

 

[_Dejan_]

As I know(Im new to ZFS) RAIDZ1 is RAID5 so 4x 4TB will give you 12TB of storage. And when you format this to some filesystem you will get 10-11TB(I didn't calculate useable space but I think that you will get something like that) so your 10.4TB in RAIDZ1(RAID 5) I think is normal...
Maybe Im miss something?

 

[sonnyj]

12TB is in line with my understanding. I'm also ok with 10.4TB as stated above. The question is why is Proxmox using 7.4TB when I'm creating a 5TB dataset on that 10.4TB?.

 

[apoc]

had a similar issue with my setup and padding overhead.
Maybe this helps:
https://forum.proxmox.com/threads/u...aidz3-pool-vs-mirrored-pool.65018/post-293908

 

[Dunuin]

Like apoc already said its padding overhead. With ashift=12, a default volblocksize of 8K and a four disk raidz1 you only get 40% of the raw capacity of usable storage for VM disks. So with 16TB of raw storage only 6.4 TB or 5.82 TiB.

Raw storage is 16TB. You loose 25% for parity so ZFS will tell you that you got 12TB of usable capacity. But you also indirectly loose 25% of raw storage when using zvols because of padding overhead. Thats because everything written to a zvol will be 150% in size. So when writing 8TB of data to a zvol it also needs to write 4 TB of empty padding blocks. So after writing 8 TB of data your pool is full.
And a ZFS pool should never be filled completely. 20% of the capacity should always be kept free or the pool becomes slow and fragments faster (and there is no option to defrag it because of Copy-on-Write). So of this 8TB you again loose 20% and only got 6.4TB of data that can be stored.

I would recommend to use a striped mirror (aka raid10) instead. You get the same 40% of usable raw capacity but a multiple of the performance, faster resilvering, better expandability and more reliability.

If you don't care about all that and want to get the most capacity out of a raidz1 you could increase the volblocksize to 16K to get 53% of the raw capacity as usable storage. Or even a volblocksize of 64K to get 58% of the raw capacity as usable VM storage.
But the higher you choose your volblocksize, the more terrible the performance will become when writing small blocks. So for example a really bad idea if you need to run databases etc.

 

[sonnyj]

Thank you very much - volblocksize did the trick. zfs create -V 5T -o volblocksize=64k now uses up 5.02T instead of the previous 7.43T.

 

[Dunuin]

Thank you very much - volblocksize did the trick. zfs create -V 5T -o volblocksize=64k now uses up 5.02T instead of the previous 7.43T.

You can create zvols that way but keep in mind that when migrating or restoring VM from backup it will create the zvols using the value as volblocksize what is set for the ZFS storage at "Datacenter -> Storage -> YourZFSPool -> Edit -> Blocksize". So if you don't change there to 64K too they will be restored/migrated with the default 8K volblocksize again.