local-zfs volume is full and I don't understand why

[f4242]

Hello,

My local-zfs volume (2 ssd in mirror) is full and I don't understand why. I have only one VM with 3 disks configured and the total size of my virtuals disks doesn't match the used space. I don't have any snapshots.

root@pve-ti1:~# zfs list
NAME                        USED  AVAIL     REFER  MOUNTPOINT
rpool                       430G   627M      104K  /rpool
rpool/ROOT                 1.48G   627M       96K  /rpool/ROOT
rpool/ROOT/pve-1           1.48G   627M     1.48G  /
rpool/data                  428G   627M       96K  /rpool/data
rpool/data/vm-3085-disk-0  47.4G  46.0G     2.07G  -
rpool/data/vm-3085-disk-1  4.13G  4.51G      239M  -
rpool/data/vm-3085-disk-2   376G  79.5G      298G  -

root@pve-ti1:~# zfs list -t snapshot
no datasets available

ZFS volume stat on the PVE GUI:





The virtuals disks configured for the unique VM hosted on that PVE server, total max size should never be greater than 46+4+365, so 415GB w/o using snapshots.



In the VM:

root@pbs:/data# LANG=C df -h
Filesystem            Size  Used Avail Use% Mounted on
[removed useless lines]
/dev/mapper/pbs-root   39G  2.6G   35G   7% /
/dev/sdb              359G  202G  139G  60% /data

I tried to fstrim my file systems inside the VM, but this didn't free up anything.

Any idea?

Thanks!

 

[f4242]

Sorry to bump the thread. Should I submit more information to help the diagnostic?

Thanks.

 

[fiona]

Hi,

Hello,

My local-zfs volume (2 ssd in mirror) is full and I don't understand why. I have only one VM with 3 disks configured and the total size of my virtuals disks doesn't match the used space. I don't have any snapshots.

root@pve-ti1:~# zfs list
NAME                        USED  AVAIL     REFER  MOUNTPOINT
rpool                       430G   627M      104K  /rpool
rpool/ROOT                 1.48G   627M       96K  /rpool/ROOT
rpool/ROOT/pve-1           1.48G   627M     1.48G  /
rpool/data                  428G   627M       96K  /rpool/data
rpool/data/vm-3085-disk-0  47.4G  46.0G     2.07G  -
rpool/data/vm-3085-disk-1  4.13G  4.51G      239M  -
rpool/data/vm-3085-disk-2   376G  79.5G      298G  -

root@pve-ti1:~# zfs list -t snapshot
no datasets available

but the size does match? 376 + 4 + 47 + = 427 and then add 1 for the fractional parts.

The virtuals disks configured for the unique VM hosted on that PVE server, total max size should never be greater than 46+4+365, so 415GB w/o using snapshots.

View attachment 33208

No, this means that this much space is available on the virtual block devices. ZFS will often need more space to represent it because of metadata/how it organizes the blocks.

 

[f4242]

Hello,

Hi,

but the size does match? 376 + 4 + 47 + = 427 and then add 1 for the fractional parts.

Yes but this doesn't reflate the real usage inside the VM. See other df output in my orignal post.

No, this means that this much space is available on the virtual block devices. ZFS will often need more space to represent it because of metadata/how it organizes the blocks.

So it's normal that zfs is using more space on my drives than the max file size of my virtual hdd? Even if the real usage on the real hdd is not completely used (about 205 GB is used in the virtual drive)? We don't talk about some megabytes, I can't use more than 50% on my storage!

Thanks!

 

[Dunuin]

I guess you are using a raidz1/2/3? There it is normal that the virtual disk (zvol) needs a multiple (like 133%/150%/200%) of the space used by the VM if your volblocksize is too low because of padding overhead.

 

[fiona]

Hello,



Yes but this doesn't reflate the real usage inside the VM. See other df output in my orignal post.

Proxmox VE does not create sparse zvols by default, but you can set this in the storage configuration (Thin provision option in Datacenter > Storage > Edit). Note that this will only take effect for newly created volumes, but you can try to do move to another storage and than back again.

So it's normal that zfs is using more space on my drives than the max file size of my virtual hdd? Even if the real usage on the real hdd is not completely used (about 205 GB is used in the virtual drive)? We don't talk about some megabytes, I can't use more than 50% on my storage!

Thanks!

But the virtual drive is not full yet. ZFS just already reserved the space.