ZFS reporting drastically different numbers

[DJFriar]

My initial research is pointing at padding overhead (and it is possible thin provisioning didn't get enabled at creation), but this also seems wildly off compared to other examples.

The drive called "zfs" is a made up of six 1.96TB SSD drives in a RAIDZ. When I go to the summary for the zfs "drive" on my node, I see 97.8% usage, 8.86TB of 9.06TB (screenshot attached). But when I go to Node -> Disks -> ZFS, it shows 9.05TB free with 2.46 TB allocated (screenshot attached). I know padding overhead can cause more space to be used than expected, but this seems wildly off. For what it is worth, my volblocksize is 16k.

I do have replication enabled as this is one of two nodes in the cluster, and the other node is reporting similarly.

My questions are:

1) What could be causing this?
2) Is there a way to see if thin provisioning is enabled on this zfs drive or not?
2a) if not, I assume I will need to copy this data to a larger drive, then remove the zfs and re-create it; then copy the data back; correct?
3) If 16k volblocksize is incorrect, what would be the better setting? We are slightly space over IOPS focused.
4) I assume I can add physical drives in a non-destructive way, right?

 

[fireon]

Is there a way to see if thin provisioning is enabled on this zfs drive or not?

If Thin provision is not activated, this is displayed as such. Right. But why it is only displayed on storage utilization and not on the Zpool, I can't tell you yet either. I have never used a Zpool without thin provision.

My theory is that the VM storage is reserved dynamically. ZFS only allocates blocks when data is actually written. Even if the storage is displayed as occupied in Proxmox, ZFS may not yet have physically occupied the storage.
But someone else can explain that better to us...

I assume I can add physical drives in a non-destructive way, right?

That depends: https://forum.proxmox.com/threads/zfs-raidz1-expanding-not-possible.127480/

2a) if not, I assume I will need to copy this data to a larger drive, then remove the zfs and re-create it; then copy the data back; correct?

I think a "zfs send" and "zfs receive" in this case did not work, because it is a problem (as it seems) not with the pool but with the vdevs (vm images). I believe the following should work (untested):

1. Backup your virtual machines with vzdump/PBS (PBS has integrated verification of the backuped images)
2. Delete all VM's on the PVE Host
3. Remove the ZFS storage pool from "Datacenter -> Storage -> <your ZFS pool>"
4. Recreate this pool but with the flag on "Thin porvision:"
5. Restore your VM's

Now everything should be fine with your proxmox storage and ZFS space.

 

[DJFriar]

That depends: https://forum.proxmox.com/threads/zfs-raidz1-expanding-not-possible.127480/


I think a "zfs send" and "zfs receive" in this case did not work, because it is a problem (as it seems) not with the pool but with the vdevs (vm images). I believe the following should work (untested):

1. Backup your virtual machines with vzdump/PBS (PBS has integrated verification of the backuped images)
2. Delete all VM's on the PVE Host
3. Remove the ZFS storage pool from "Datacenter -> Storage -> <your ZFS pool>"
4. Recreate this pool but with the flag on "Thin porvision:"
5. Restore your VM's

Now everything should be fine with your proxmox storage and ZFS space.

I do have replication enabled, so in theory I could just migrate them to the other node, then erase and recreate the zfs pool with thin provisioning on, then migrate them back over; then repeat the process the other way to do the other node, right?