ZFS used and free space

MaxZhurkin

New Member
Sep 18, 2023
1
0
1
Hello!
Is there any article on the net that would clearly describe how to find out how much space is actually occupied by various objects in ZFS, such as FS, ZVOL and how to find out the actual available space taking into account quotas?
From other posts I understand that in my case at least 25% is lost in overhead expenses (ashift=12 (so 4K "sectors") and volblocksize=16K) not counting parity data.
There are snapshots skipped everywhere, the total number of snapshots is close to a thousand.
One gets the impression that there is some kind of reservation involved here.

Code:
# zpool list -v
NAME                         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
some-zfs-pool                 43.6T  3.67T  40.0T        -         -    10%     8%  2.00x  ONLINE  -
raidz2-0                  14.5T  1.19T  13.4T        -         -    10%  8.19%      -    ONLINE
sdi                     1.82T      -      -        -         -      -      -      -    ONLINE
...
raidz2-1                  14.5T  1.24T  13.3T        -         -    10%  8.52%      -    ONLINE
sdq                     1.82T      -      -        -         -      -      -      -    ONLINE
...
raidz2-2                  14.5T  1.23T  13.3T        -         -    10%  8.47%      -    ONLINE
sdy                     1.82T      -      -        -         -      -      -      -    ONLINE
...
logs                            -      -      -        -         -      -      -      -  -
nvme0n1                    238G   468M   238G        -         -     0%  0.19%      -    ONLINE
cache                           -      -      -        -         -      -      -      -  -
nvme1n1                    233G  53.6G   179G        -         -     0%  23.0%      -    ONLINE

Code:
# zfs list -o space |grep -E 'storage|NAME'
NAME                                    AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
some-zfs-pool/vm-storage                  1.40T  4.60T        0B    205K             0B      4.60T
some-zfs-pool/vm-storage/base-100-disk-0  1.43T  58.7G     5.17G   18.8G          34.6G         0B
some-zfs-pool/vm-storage/base-104-disk-0  1.43T  39.5G     1.05G   3.80G          34.6G         0B
some-zfs-pool/vm-storage/vm-101-disk-0    1.76T   964G      249G    347G           368G         0B
some-zfs-pool/vm-storage/vm-102-disk-0    1.43T  64.1G     23.6G   8.30G          32.2G         0B
some-zfs-pool/vm-storage/vm-103-disk-0    1.60T   757G      344G    200G           212G         0B
some-zfs-pool/vm-storage/vm-105-disk-0    1.50T   123G     1.50G   11.5G           110G         0B
some-zfs-pool/vm-storage/vm-106-disk-0    1.71T   606G     13.4G    268G           325G         0B
some-zfs-pool/vm-storage/vm-106-disk-1    1.71T   332G     85.9M   7.29G           325G         0B
some-zfs-pool/vm-storage/vm-107-disk-0    1.44T  47.8G      476M   4.04G          43.3G         0B
some-zfs-pool/vm-storage/vm-108-disk-0    1.44T  47.1G     16.4M   3.81G          43.3G         0B
some-zfs-pool/vm-storage/vm-109-disk-0    1.44T  47.1G     19.6M   3.81G          43.3G         0B
some-zfs-pool/vm-storage/vm-110-disk-0    1.43T  38.5G     19.3M   3.81G          34.6G         0B
some-zfs-pool/vm-storage/vm-111-disk-0    1.44T  47.1G     20.1M   3.81G          43.3G         0B
some-zfs-pool/vm-storage/vm-112-disk-0    1.44T  47.1G     17.1M   3.80G          43.3G         0B
some-zfs-pool/vm-storage/vm-113-disk-0    1.60T   380G     16.1G    160G           204G         0B
some-zfs-pool/vm-storage/vm-113-disk-1    1.64T   498G     34.3G    217G           247G         0B
some-zfs-pool/vm-storage/vm-114-disk-0    1.52T   130G        0B    119K           130G         0B
some-zfs-pool/vm-storage/vm-114-disk-1    1.52T   259G     15.0G    114G           130G         0B
some-zfs-pool/vm-storage/vm-114-disk-2    1.51T   190G     1.85G   74.3G           114G         0B
some-zfs-pool/vm-storage/vm-115-disk-0    1.43T  38.5G     31.8M   3.80G          34.6G         0B

Code:
# zpool get ashift
NAME         PROPERTY  VALUE   SOURCE
some-zfs-pool  ashift    12      local

Code:
# zfs get volblocksize |grep -v autosnap |grep -E 'storage|NAME'
NAME                                                                         PROPERTY      VALUE     SOURCE
some-zfs-pool/vm-storage                                                       volblocksize  -         -
some-zfs-pool/vm-storage/base-100-disk-0                                       volblocksize  16K       -
some-zfs-pool/vm-storage/base-100-disk-0@__base__                              volblocksize  -         -
some-zfs-pool/vm-storage/base-104-disk-0                                       volblocksize  16K       -
some-zfs-pool/vm-storage/base-104-disk-0@__base__                              volblocksize  -         -
some-zfs-pool/vm-storage/vm-101-disk-0                                         volblocksize  16K       -
...
some-zfs-pool/vm-storage/vm-115-disk-0                                         volblocksize  16K       -

Code:
# zfs get quota |grep -v 'quota     -      -' |grep -E 'storage|NAME'
NAME                                                                         PROPERTY  VALUE  SOURCE
some-zfs-pool/vm-storage                                                       quota     6T     local
 
Last edited:
the ZFS manpages are a good starting point:

man zpoolprops
man zfsprops

note that zpool gives you the raw/disk/vdev view, while zfs gives you the logical/dataset/volume view.

there are lots of components involved:
- compression
- ashift/raidz overhead
- reservations and quotas
- snapshots
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!