[SOLVED] ZFS Pool "backup" 100% Full, GC fails

[t0mc@]

Hi everybody,

at a PBS the backup storage, which is a ZFS, is completely full, so the GC Job fails:


In the Server all physical ports are used so we can't simply add more HDDs for extending the pool.
In theory we just need some few megs of space so the GC Job can do it's work.. we even tried to move away one or two dirs under .chunks (for later moving them back of course), but the space remains completely full.

What can we do to get a little more free space in the ZFS?
And we are wondering what this column is about:

It sais "133.76GB" available .. but where? And how could we make it usable to the backup pool?
This is what "df" sais:



Any hint? Thx in advance!

 

[t0mc@]

After googling around, another thing we tried was this:
-> copy an old dir under .chunks to another FS
-> "cat /dev/null >..." to all files in that dir
-> rm all files in that dir

Didn't help either, FS remains 100% full... as a plus what we noticed and is even more confusing:
Before this step we did a df which sais this:

Filesystem             1K-blocks        Used Available Use% Mounted on
backup               15437977344 15437977344         0 100% /mnt/datastore/backup

After the "cat ..." and "rm..." df sais this:

Filesystem             1K-blocks        Used Available Use% Mounted on
backup               15437739520 15437739520         0 100% /mnt/datastore/backup

The FS completely shrunk?

 

[dcsapak]

what does

zfs list
zpool list

show ?

 

[t0mc@]

root@pbs:~# zfs list
NAME     USED  AVAIL     REFER  MOUNTPOINT
backup  14.4T     0B     14.4T  /mnt/datastore/backup
root@pbs:~# zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
backup  14.5T  14.4T   125G        -         -    89%    99%  1.00x    ONLINE  -

 

[dcsapak]

ok for some reason zfs says you have '0B' available for the dataset, do you have any snapshots?
(zfs list -t all)

 

[t0mc@]

Nope:

root@pbs:~# zfs list -t all
NAME     USED  AVAIL     REFER  MOUNTPOINT
backup  14.4T     0B     14.4T  /mnt/datastore/backup
root@pbs:~#

 

[t0mc@]

Nobody with an idea?

 

[t0mc@]

I just noticed this:

root@pbs:/mnt/datastore/backup# df -i
Filesystem             Inodes   IUsed    IFree IUse% Mounted on
backup                6828558 6828558        0  100% /mnt/datastore/backup

Could this be an explanation?
I'm wondering if / that ZFS really also has inodes like ext4... How can one influence it?

 

[dcsapak]

mhmm that seems weird, but can you post the output of

zfs get all

?

 

[t0mc@]

This PBS is our Backup Target for all productive / critical VMs of a connected productive PVE Cluster. So as we urgently need it running again, and no one seems to have an idea we did the following in the meantime:

-> Moved away ~30-40 of the oldest directories in ".chunks" to another FS and so gettet 15GB of free space
-> Started the GC Job again (which of course now is warning about missing chunks) but now it seems to come through, freeing much more space

So at this moment GC is running, freeing more and more. Actually df commands look like this:

root@pbs:~# df -hi
Filesystem           Inodes IUsed IFree IUse% Mounted on
backup                 220M  6.5M  214M    3% /mnt/datastore/backup

Notice the HUGE increasing of available inodes now (the last time it was ~6.8M, now it is 220M(!) )

root@pbs:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
backup                 15T   15T  111G 100% /mnt/datastore/backup

And here - for completness - the output of "zfs get all":

root@pbs:~# zfs get all
NAME    PROPERTY              VALUE                  SOURCE
backup  type                  filesystem             -
backup  creation              Sun May 15 20:23 2022  -
backup  used                  14.3T                  -
backup  available             113G                   -
backup  referenced            14.3T                  -
backup  compressratio         1.02x                  -
backup  mounted               yes                    -
backup  quota                 none                   default
backup  reservation           none                   default
backup  recordsize            128K                   default
backup  mountpoint            /mnt/datastore/backup  local
backup  sharenfs              off                    default
backup  checksum              on                     default
backup  compression           on                     local
backup  atime                 on                     default
backup  devices               on                     default
backup  exec                  on                     default
backup  setuid                on                     default
backup  readonly              off                    default
backup  zoned                 off                    default
backup  snapdir               hidden                 default
backup  aclmode               discard                default
backup  aclinherit            restricted             default
backup  createtxg             1                      -
backup  canmount              on                     default
backup  xattr                 on                     default
backup  copies                1                      default
backup  version               5                      -
backup  utf8only              off                    -
backup  normalization         none                   -
backup  casesensitivity       sensitive              -
backup  vscan                 off                    default
backup  nbmand                off                    default
backup  sharesmb              off                    default
backup  refquota              none                   default
backup  refreservation        none                   default
backup  guid                  14939185443403254439   -
backup  primarycache          all                    default
backup  secondarycache        all                    default
backup  usedbysnapshots       0B                     -
backup  usedbydataset         14.3T                  -
backup  usedbychildren        670M                   -
backup  usedbyrefreservation  0B                     -
backup  logbias               latency                default
backup  objsetid              54                     -
backup  dedup                 off                    local
backup  mlslabel              none                   default
backup  sync                  standard               default
backup  dnodesize             legacy                 default
backup  refcompressratio      1.02x                  -
backup  written               14.3T                  -
backup  logicalused           14.6T                  -
backup  logicalreferenced     14.6T                  -
backup  volmode               default                default
backup  filesystem_limit      none                   default
backup  snapshot_limit        none                   default
backup  filesystem_count      none                   default
backup  snapshot_count        none                   default
backup  snapdev               hidden                 default
backup  acltype               off                    default
backup  context               none                   default
backup  fscontext             none                   default
backup  defcontext            none                   default
backup  rootcontext           none                   default
backup  relatime              on                     local
backup  redundant_metadata    all                    default
backup  overlay               on                     default
backup  encryption            off                    default
backup  keylocation           none                   default
backup  keyformat             none                   default
backup  pbkdf2iters           0                      default
backup  special_small_blocks  0                      default

When the currently running GC Job is finished successfully (hopefully), we will do this next:
-> Move the former moved away chunk dirs back to the .chunks folder
-> Let GC Job run again

 

[Max Carrara]

Taken from man 4 zfs:

spa_slop_shift=5 (1/32nd) (int)
Normally, we don't allow the last 3.2% (1/2^spa_slop_shift) of space in the pool to be consumed. This ensures that we don't run the pool completely out of space, due to unaccounted changes (e.g. to the MOS). It also limits the worst-case time to allocate space. If we have less than this amount of free space, most ZPL operations (e.g. write, create) will return ENOSPC.

Credit goes to @mira for finding that.

For what's worth, I also managed to replicate that behaviour within a VM, on a simple mirror pool with two 8 GiB drives.

Steps to reproduce:

# assuming /dev/sdb and /dev/sdc are available and equally-sized
zpool create tank mirror /dev/sdb /dev/sdc
zfs create tank/test
cd /tank/test
dd if=/dev/random of=some_file bs=4M

dd will abort automatically when the zpool is full.

zpool list and zfs list will then show similar results as with your pool:

root@zfs-shenanigans:/tank/test# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank  7.50G  7.27G   236M        -         -    58%    96%  1.00x    ONLINE  -
root@zfs-shenanigans:/tank/test# zfs list
NAME        USED  AVAIL     REFER  MOUNTPOINT
tank       7.27G     0B       24K  /tank
tank/test  7.27G     0B     7.27G  /tank/test

In other words: Your pool is effectively full.

 

[t0mc@]

@Max Carrara Thx a lot for this finding in the docs. Now it gets clearer why there is some space "FREE" but "No space left".

As can be read, in the meantime we were able to initially free enough space to make GC Job run again (which is still running at the time of writing).