Where does the space in zfs go?

vadimushka

Member
May 26, 2020
7
1
23
36
I have 2 nodes in the cluster. I have set up zfs storage on 2 servers.
Raid10, 4 16Tb disks

Code:
pool: vms
state: ONLINE
config:

        NAME                                   STATE     READ WRITE CKSUM
        vms                                    ONLINE       0     0     0
          mirror-0                             ONLINE       0     0     0
            ata-ST16000NM000J-2TW103_WRS0LKCN  ONLINE       0     0     0
            ata-ST16000NM000J-2TW103_WRS0LKEY  ONLINE       0     0     0
          mirror-1                             ONLINE       0     0     0
            ata-ST16000NM000J-2TW103_WRS0LJXS  ONLINE       0     0     0
            ata-ST16000NM000J-2TW103_WRS0LKCB  ONLINE       0     0     0


Code:
root@pve200:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool   444G  18.5G   426G        -         -     4%     4%  1.00x    ONLINE  -
vms    29.1T  3.63T  25.5T        -         -     0%    12%  1.00x    ONLINE  -

Code:
root@pve200:~# zfs get all vms
NAME  PROPERTY              VALUE                  SOURCE
vms   type                  filesystem             -
vms   creation              Tue Feb 18 11:08 2025  -
vms   used                  25.5T                  -
vms   available             3.51T                  -
vms   referenced            96K                    -
vms   compressratio         1.01x                  -
vms   mounted               yes                    -
vms   quota                 none                   default
vms   reservation           none                   default
vms   recordsize            128K                   default
vms   mountpoint            /vms                   default
vms   sharenfs              off                    default
vms   checksum              on                     default
vms   compression           on                     local
vms   atime                 on                     default
vms   devices               on                     default
vms   exec                  on                     default
vms   setuid                on                     default
vms   readonly              off                    default
vms   zoned                 off                    default
vms   snapdir               hidden                 default
vms   aclmode               discard                default
vms   aclinherit            restricted             default
vms   createtxg             1                      -
vms   canmount              on                     default
vms   xattr                 on                     default
vms   copies                1                      default
vms   version               5                      -
vms   utf8only              off                    -
vms   normalization         none                   -
vms   casesensitivity       sensitive              -
vms   vscan                 off                    default
vms   nbmand                off                    default
vms   sharesmb              off                    default
vms   refquota              none                   default
vms   refreservation        none                   default
vms   guid                  4066298117400478531    -
vms   primarycache          all                    default
vms   secondarycache        all                    default
vms   usedbysnapshots       0B                     -
vms   usedbydataset         96K                    -
vms   usedbychildren        25.5T                  -
vms   usedbyrefreservation  0B                     -
vms   logbias               latency                default
vms   objsetid              54                     -
vms   dedup                 off                    default
vms   mlslabel              none                   default
vms   sync                  standard               default
vms   dnodesize             legacy                 default
vms   refcompressratio      1.00x                  -
vms   written               96K                    -
vms   logicalused           3.68T                  -
vms   logicalreferenced     42K                    -
vms   volmode               default                default
vms   filesystem_limit      none                   default
vms   snapshot_limit        none                   default
vms   filesystem_count      none                   default
vms   snapshot_count        none                   default
vms   snapdev               hidden                 default
vms   acltype               off                    default
vms   context               none                   default
vms   fscontext             none                   default
vms   defcontext            none                   default
vms   rootcontext           none                   default
vms   relatime              on                     default
vms   redundant_metadata    all                    default
vms   overlay               on                     default
vms   encryption            off                    default
vms   keylocation           none                   default
vms   keyformat             none                   default
vms   pbkdf2iters           0                      default
vms   special_small_blocks  0                      default
vms   prefetch              all                    default


Installed the 2nd vm:
vm-100 - 6TB
vm-103 - 16Tb

I have set up replication.

the size of vms/vm-100-disk-0 and vms/vm-103-disk-0 is constantly increasing, can something be done about it?
A day ago it was:

Code:
root@pve200:~# zfs list -o space
NAME                      AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool                      401G  28.9G        0B    104K             0B      28.9G
rpool/ROOT                 401G  11.3G        0B     96K             0B      11.3G
rpool/ROOT/pve-1           401G  11.3G        0B   11.3G             0B         0B
rpool/data                 401G  10.4G        0B     96K             0B      10.4G
rpool/data/vm-104-disk-0   401G    96K        0B     96K             0B         0B
rpool/data/vm-104-disk-1   401G  10.4G        0B   10.4G             0B         0B
rpool/data/vm-104-disk-2   401G    64K        0B     64K             0B         0B
rpool/var-lib-vz           401G  7.15G        0B   7.15G             0B         0B
vms                       3.90T  25.1T        0B     96K             0B      25.1T
vms/vm-100-disk-0         9.85T  8.04T        0B   2.08T          5.95T         0B
vms/vm-103-disk-0         19.8T  17.0T     8.44G   1.17T          15.9T         0B

Now:
Code:
vms                3.51T  25.5T        0B     96K             0B      25.5T
vms/vm-100-disk-0  9.47T  8.39T        0B   2.44T          5.95T         0B
vms/vm-103-disk-0  19.4T  17.1T      333M   1.20T          15.9T         0B

Code:
root@pve200:~# zfs get all vms/vm-100-disk-0
NAME               PROPERTY              VALUE                     SOURCE
vms/vm-100-disk-0  type                  volume                    -
vms/vm-100-disk-0  creation              Tue Feb 18 11:10 2025     -
vms/vm-100-disk-0  used                  8.39T                     -
vms/vm-100-disk-0  available             9.47T                     -
vms/vm-100-disk-0  referenced            2.44T                     -
vms/vm-100-disk-0  compressratio         1.01x                     -
vms/vm-100-disk-0  reservation           none                      default
vms/vm-100-disk-0  volsize               5.86T                     local
vms/vm-100-disk-0  volblocksize          16K                       default
vms/vm-100-disk-0  checksum              on                        default
vms/vm-100-disk-0  compression           on                        inherited from vms
vms/vm-100-disk-0  readonly              off                       default
vms/vm-100-disk-0  createtxg             27                        -
vms/vm-100-disk-0  copies                1                         default
vms/vm-100-disk-0  refreservation        5.95T                     received
vms/vm-100-disk-0  guid                  541084260817496041        -
vms/vm-100-disk-0  primarycache          all                       default
vms/vm-100-disk-0  secondarycache        all                       default
vms/vm-100-disk-0  usedbysnapshots       0B                        -
vms/vm-100-disk-0  usedbydataset         2.44T                     -
vms/vm-100-disk-0  usedbychildren        0B                        -
vms/vm-100-disk-0  usedbyrefreservation  5.95T                     -
vms/vm-100-disk-0  logbias               latency                   default
vms/vm-100-disk-0  objsetid              516                       -
vms/vm-100-disk-0  dedup                 off                       default
vms/vm-100-disk-0  mlslabel              none                      default
vms/vm-100-disk-0  sync                  standard                  default
vms/vm-100-disk-0  refcompressratio      1.01x                     -
vms/vm-100-disk-0  written               0                         -
vms/vm-100-disk-0  logicalused           2.45T                     -
vms/vm-100-disk-0  logicalreferenced     2.45T                     -
vms/vm-100-disk-0  volmode               default                   default
vms/vm-100-disk-0  snapshot_limit        none                      default
vms/vm-100-disk-0  snapshot_count        none                      default
vms/vm-100-disk-0  snapdev               hidden                    default
vms/vm-100-disk-0  context               none                      default
vms/vm-100-disk-0  fscontext             none                      default
vms/vm-100-disk-0  defcontext            none                      default
vms/vm-100-disk-0  rootcontext           none                      default
vms/vm-100-disk-0  redundant_metadata    all                       default
vms/vm-100-disk-0  encryption            off                       default
vms/vm-100-disk-0  keylocation           none                      default
vms/vm-100-disk-0  keyformat             none                      default
vms/vm-100-disk-0  pbkdf2iters           0                         default
vms/vm-100-disk-0  snapshots_changed     Sat Feb 22 14:15:06 2025  -
vms/vm-100-disk-0  prefetch              all                       default

Code:
root@pve200:~# zfs get all vms/vm-103-disk-0
NAME               PROPERTY              VALUE                     SOURCE
vms/vm-103-disk-0  type                  volume                    -
vms/vm-103-disk-0  creation              Tue Feb 18 21:37 2025     -
vms/vm-103-disk-0  used                  17.1T                     -
vms/vm-103-disk-0  available             19.4T                     -
vms/vm-103-disk-0  referenced            1.20T                     -
vms/vm-103-disk-0  compressratio         1.02x                     -
vms/vm-103-disk-0  reservation           none                      default
vms/vm-103-disk-0  volsize               15.6T                     local
vms/vm-103-disk-0  volblocksize          16K                       default
vms/vm-103-disk-0  checksum              on                        default
vms/vm-103-disk-0  compression           on                        inherited from vms
vms/vm-103-disk-0  readonly              off                       default
vms/vm-103-disk-0  createtxg             7482                      -
vms/vm-103-disk-0  copies                1                         default
vms/vm-103-disk-0  refreservation        15.9T                     local
vms/vm-103-disk-0  guid                  1800627205636521664       -
vms/vm-103-disk-0  primarycache          all                       default
vms/vm-103-disk-0  secondarycache        all                       default
vms/vm-103-disk-0  usedbysnapshots       633M                      -
vms/vm-103-disk-0  usedbydataset         1.20T                     -
vms/vm-103-disk-0  usedbychildren        0B                        -
vms/vm-103-disk-0  usedbyrefreservation  15.9T                     -
vms/vm-103-disk-0  logbias               latency                   default
vms/vm-103-disk-0  objsetid              2782                      -
vms/vm-103-disk-0  dedup                 off                       default
vms/vm-103-disk-0  mlslabel              none                      default
vms/vm-103-disk-0  sync                  standard                  default
vms/vm-103-disk-0  refcompressratio      1.02x                     -
vms/vm-103-disk-0  written               618M                      -
vms/vm-103-disk-0  logicalused           1.23T                     -
vms/vm-103-disk-0  logicalreferenced     1.23T                     -
vms/vm-103-disk-0  volmode               default                   default
vms/vm-103-disk-0  snapshot_limit        none                      default
vms/vm-103-disk-0  snapshot_count        none                      default
vms/vm-103-disk-0  snapdev               hidden                    default
vms/vm-103-disk-0  context               none                      default
vms/vm-103-disk-0  fscontext             none                      default
vms/vm-103-disk-0  defcontext            none                      default
vms/vm-103-disk-0  rootcontext           none                      default
vms/vm-103-disk-0  redundant_metadata    all                       default
vms/vm-103-disk-0  encryption            off                       default
vms/vm-103-disk-0  keylocation           none                      default
vms/vm-103-disk-0  keyformat             none                      default
vms/vm-103-disk-0  pbkdf2iters           0                         default
vms/vm-103-disk-0  snapshots_changed     Sat Feb 22 13:52:02 2025  -
vms/vm-103-disk-0  prefetch              all                       default
 
Last edited: