ZFS IO-ERROR Free Space Issues

[TheKingOfRainyLands]

I have a 7 disk zfs raidz2 made up of 7 8tb drives.
I created this back in February, but have recently been running into issues with it.
Every once and a while my primary storage vm has been crashing due to an io-error.

As you can see the free space has slowly gone down over time.
I have not been creating more virtual disks, as I originally intended to have 2TB of free space for later projects.
I have had to slowly move disks off of this datapool to another disk(one that is not redundant)in order to reboot that VM.
I am down to my last one I have been able to move off.
I don't understand exactly why this is happening.
root@proxmoxMachine:~# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT datapool 50.9T 49.0T 1.92T root@proxmoxMachine:~# zfs list NAME USED AVAIL REFER MOUNTPOINT datapool 32.8T 33.0G 192K /datapool datapool/vm-100-disk-0 51.6G 75.3G 9.24G - datapool/vm-100-disk-1 32.6T 33.0G 32.6T - datapool/vm-104-disk-0 103G 116G 19.9G - datapool/vm-108-disk-0 103G 99.9G 36.3G -

I have no idea what I need to do to fix this.
I was hoping that I would have 32TB+ of space, but at this rate im careening towards even less than that.
I really don't want to have to resize "vm-100-disk-1" because thats the one with everything off it and I fear data loss if I play with the partition on the VM.

Im honestly annoyed with this, I have no idea what to do and where I went wrong, I suppose I assumed that zfs wouldn't affect space
I fear I am typing this on borrowed time, as its only a matter of time till I can no longer move any more disks off.

 

[guletz]

Hi,

Try to check on your virtual disk if you have chose chache=none. Also check in the vm if you can run a trim command (for linux os is: fstrim -va).

Zfs without cache=none + fstrim can not know that inside a vm you delete / free some blocks.

And it also possible to have snapshots of your VMs that use the zfs space.

Good luck / Bafta!

 

[TheKingOfRainyLands]

This is currently what I have it set as.
I should change that cache setting to something where cache is enabled, correct?
Thank you for your help this has been driving me crazy for a while

 

[TheKingOfRainyLands]

From reading the wiki I see conflicting comments about whether I should use cache "cache=none seems to be the best performance and is the default since Proxmox 2.X."

 

[Dunuin]

If you don't set the checkbox "Discard" you can't use Thin Provisioning and TRIM doesn't work.

Also your seem to use snapshots: "datapool/vm-104-disk-0 103G 116G 19.9G"
"REFER" (19,9G) is the size without childs/snapshots and "USED" (103G) is the size with snapshots. So your Snapshots consume 83GB of data. Make sure you delete old snapshots regularily. They grow with every day the VM is running.

 

[apoc]

What segmentsize/chunksize did you use when creating the zvol for the VM?