zfs pool different sizes reported

[m_l_s]

Hey, I hope anyone might be able to help and point me in the right direction to solve the following issue.

I have a zfs-pool in my machine which I use for mass storage. While I knew I was running low on storage, I expected to still have a few gigs available. But when I tried to rename a file on the pool I got an out of Storage error: (/Data2 is the pool MP)

$ touch /Data2/tst
touch: cannot touch '/Data2/tst': No space left on device

Upon looking in the Web-interface it still said I have ~135Gb left, so I took a closer look and discovered the following:

$ df -h /Data2/

Filesystem      Size  Used Avail Use% Mounted on

Data2           7.9T  7.9T     0 100% /Data2

$ zpool get capacity,size,health,fragmentation

NAME   PROPERTY       VALUE     SOURCE

Data2  capacity       98%       -

Data2  size           10.9T     -

Data2  health         ONLINE    -

Data2  fragmentation  34%       -

Somehow du and zpool report significantly different capacities and uses. I expect zpool/the web-interface to be correct, because the Pool consists of 2 12T drives and was created with exactly those drives.

How can I maybe use my last bit of storage?

 

[Pifouney]

zfs = RAID
RAID = metadatas
zpool get = RAW additionned disk capacity ofd the raid.
du & zfs list/get = usable disk capacity ( iin the raid)
GUI zpool report = zpool get

 

[waltar]

zfs reserve a little bit space for recovering from a full pool as deleting itself needs new space as cow filesystem for new dir timestamps. If there would be no reserve it would be unpossible to delete any files anymore.
See https://openzfs.github.io/openzfs-docs/Performance and Tuning/Module Parameters.html :
cat /sys/module/zfs/parameters/spa_slop_shift # would be around 3
echo 2 > /sys/module/zfs/parameters/spa_slop_shift
Remove old unused data in pool AND set again to orig. value as seen with cat before !!

 

[esi_y]

ZFS is generally weird in how space is reported, especially on RAIDZs.

And then there's the mysterious issues like ... taking up space for no one:
https://forum.proxmox.com/threads/w...reported-avail-and-refreserv-on-zvols.151874/

 

[waltar]

And then there's the mysterious issues like ...

omg zfs ... but lvm and lvm-thin combi isn't as much easier in space understanding even with snapshot usage, for block storage just think about complex mathematics and you are in your element to follow

 

[m_l_s]

zfs = RAID
RAID = metadatas
zpool get = RAW additionned disk capacity ofd the raid.
du & zfs list/get = usable disk capacity ( iin the raid)
GUI zpool report = zpool get

Thank you for your answer

Althought I must say, I did not expect the zfs overhead to be that big .
Plus I would have thought the free space would at least match.

 

[leesteken]

oh wow, I did not expect the zfs overhead to be that big .

It's not uncommon that people get really disappointed (with performance and usable space) about RAIDz1/2 because they expect it to be the same as RAID5/6 (which it is not, as multiple threads on this forum will show in detail).

 

[esi_y]

It's not uncommon that people get really disappointed (with performance and usable space) about RAIDz1/2 because they expect it to be the same as RAID5/6 (which it is not, as multiple threads on this forum will show in detail).

Do you mind referencing one? Not that I disagree, but I did not see it discussed here.

 

[leesteken]

Do you mind referencing one? Not that I disagree, but I did not see it discussed here.

Hmm, it's often me just pointing out they're different: https://forum.proxmox.com/search/78...t=post&c[child_nodes]=1&c[nodes][0]=16&o=date

Often people complain about ZFS being slow (for VMs, which it actually is, because of all the features it provides) because they assume RAID5+BBU is the same and don't know enough (or spend enough money) to make ZFS perform well.

ZFS has a lot of metadata and the volblocksize (used for VM virtual disks) usually is not a good fit for the number of disks, which give a lot of write amplification (which is never faster then one drive). A RAID5 controller with a battery can cache sync writes and is therefore much faster then ZFS (with additional sync writes for metadata) when people use drives that don't already do this themselves (like enterprise SSDs using PLP). Since the IOPS is like a single disk, these kinds of RAIDs are not well suited for the often random/small IOPS of VMs.

 

[esi_y]

Hmm, it's often me just pointing out they're different: https://forum.proxmox.com/search/78...t=post&c[child_nodes]=1&c[nodes][0]=16&o=date

Indeed. But at least someone, when I browse through the results, it brings me back to my confusion about what people discuss (and expect). It starts with nomenclature (apparently that only bother myself [1]).
https://forum.proxmox.com/search/78...t=post&c[child_nodes]=1&c[nodes][0]=16&o=date

Often people complain about ZFS being slow (for VMs, which it actually is, because of all the features it provides) because they assume RAID5+BBU is the same and don't know enough (or spend enough money) to make ZFS perform well.

I have another issue with this on this forum. Often, it's a homelabber asking, they get responses which are not practical for follow (the recommended hw does not exist [2]). But those are not comparing with HW RAIDs, they compare with non-COW ordinary ext4 install, typically.

ZFS has a lot of metadata and the volblocksize (used for VM virtual disks) usually is not a good fit for the number of disks, which give a lot of write amplification (which is never faster then one drive).

This is true for anything that adds checksumming and is copy-on-write on top of it.

A RAID5 controller with a battery can cache sync writes and is therefore much faster then ZFS (with additional sync writes for metadata) when people use drives that don't already do this themselves (like enterprise SSDs using PLP).

I actually believe PLP in terms of saving writes time is negligible here.

Since the IOPS is like a single disk, these kinds of RAIDs are not well suited for the often random/small IOPS of VMs.

I have never had any issues with ZFS per se until I discovered on this forum people seem to use it for everything, then wonder. It's a price to pay to make technology accessible with a single click.

[1] https://forum.proxmox.com/threads/raid0-1-10-do-not-exist-in-zfs.156143/
[2] https://forum.proxmox.com/threads/performance-comparison-between-zfs-and-lvm.124295/#post-714219

 

[m_l_s]

It's not uncommon that people get really disappointed (with performance and usable space) about RAIDz1/2 because they expect it to be the same as RAID5/6 (which it is not, as multiple threads on this forum will show in detail).

Yes, I did expect it to perform similar to raid 1 as i am ruining a mirrored pool.
But I have to say the compression makes up for the size difference and I did not expect it to be that good (I copied all files over from the zfs pool (Data2) to an ext4-formated Drive (mounted to /Data0) to set up another pool on the original drives and the difference is a lot)

root@Proxmox:~# df -h /Data0/ /Data2/
Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/Data3   15T   11T  2.9T  80% /Data0
Data2              8.1T  7.9T  216G  98% /Data2

I am actually going to set up a raidZ1 now (with all three disks involved) so I'll see how that goes xD. But performance (esp. for small iops) is not that much of a concern as its just for mass storage, not used for boot-disks.