ZFS Raid showing incorrect usage?

[baron164]

I have a ZFS Raid volume on each of my hosts and when I look at the Node's ZFS Window I see the correct utilization. To be clear, each host has 12 drives, configured in a raid 10 using ZFS. I named them the same due to what I saw was a requirement for replication to function in a cluster.



However, if I look at the storage object it shows me almost double the usage which doesn't make any sense.




I connected to the host using SSH and tried to run df -h to see if I could figure out why it's reporting this way, however df -h does not show me the RAID10 volume properly.



Can anyone tell me what's going on and how to fix it? Backups are not going to this volume, though I do have Replication setup to replicate the hosts from one host to another. I also checked and did not see any snapshots on any of the VMs. I'm hoping this is just some kind of statistics error that can be easily fixed.

 

[baron164]

I ran zpool list from an SSH Session and it too is showing me the correct usage.



So it only appears to be the Disk object listed with the VMs that shows incorrect information.

 

[Dunuin]

I connected to the host using SSH and tried to run df -h to see if I could figure out why it's reporting this way, however df -h does not show me the RAID10 volume properly.

Check zpool status, zpool list -v and zfs list -o space to see whats going on. The "zpool" command got the block device POV and isn't accounting for parity or quotas while the "zfs" command, with its filesystem POV, is.

 

[baron164]

Ran zfs list -o space to view snapshot information and I'm seeing a more complete picture.



I removed replication for vm-109 and the usage reported for the local Raid10 dropped.


Re-ran zfs list -o space and saw that the USEDREFRESERV value for vm-109 dropped as well.



Is this expected behavior for replication? And is there a way to mitigate this behavior so I 'm not utilizing all of my disk space?

 

[Dunuin]

While there are some snapshots those don't consume that much space. The bigger problem is the refreservation which would indicate that you either didn't checked the "thin" checkbox when creating that ZFS storage so everything would be thick-provisioned or you didn't set up discard/TRIM properly for your guests. Every guestOS of a VM needs to report deleted blocks by sending TRIM command which means each guestOS needs to mount filesystem using the "discard" option or run fstrim -a regularly. Trimming will also not work in case you forgot to check the "discard" checkbox when creating the VMs virtual disk.

 

[baron164]

While there are some snapshots those don't consume that much space. The bigger problem is the refreservation which would indicate that you either didn't checked the "thin" checkbox when creating that ZFS storage so everything would be thick-provisioned or you didn't set up discard/TRIM properly for your guests. Every guestOS of a VM needs to report deleted blocks by sending TRIM command which means each guestOS needs to mount filesystem using the "discard" option or run fstrim -a regularly. Trimming will also not work in case you forgot to check the "discard" checkbox when create the VMs virtual disk.

Ok, I did not do either of those things. I have now gone back and checked the "Thin provision" box on the ZFS Raid10 storage objects. I have also checked the "discard" checkbox on each of the VM's virtual disks. Is there anything else I should do?

 

[Dunuin]

Is there anything else I should do?

Like already said, checking "discard" isn't enough. You have to set up discard/fstrim in every VMs guestOS to actively report deleted blocks down the storage chain so ZFS knows what blocks got deleted to free up that space. How that is done depends on the OS your VM is running.
See this in case you are not familiar what TRIM does: https://en.wikipedia.org/wiki/Trim_(computing)

 

[baron164]

The VM's I'm running all have OS's new enough where TRIM is enabled by default. I did go through and check them to confirm and it looks like they all have it enabled.