ZFS quota ignored (?) - pbs backup storage full

[mhert]

Good (?) morning.

Over the weekend our pbs storage runs out of space.

As I already had set a quota because of similar problems in the past, I wanted to change it for running a GC.

root@bk01:~# zfs set quota=14.4T local_backupstorage_bk01
cannot set property for 'local_backupstorage_bk01': size is less than current used or reserved space

OK. Thougt it was 14.3T before. Short check:

root@bk01:~# zfs list local_backupstorage_bk01
NAME USED AVAIL REFER MOUNTPOINT
local_backupstorage_bk01 14.4T 0B 14.4T /mnt/datastore/local_backupstorage_bk01
root@bk01:~#

OK. Don't Panic. Next try:

root@iser-bk01:~# zfs set quota=14.5T local_backupstorage_bk01
root@iser-bk01:~#

Better this time (no error). But output of zfs list stays the same (AVAIL):

root@bk01:~# zfs list local_backupstorage_bk01
NAME USED AVAIL REFER MOUNTPOINT
local_backupstorage_bk01 14.4T 0B 14.4T /mnt/datastore/local_backupstorage_bk01
root@bk01:~#





The volume is only for pbs backups of the cluster.


Can somebody explain to me in detail what went wrong? And how could I change this temporarily to complete a GC?

 

[mhert]

Addition:

root@bk01:/mnt/datastore/local_backupstorage_bk01# du -shx .chunks/
15T .chunks/
root@bk01:/mnt/datastore/local_backupstorage_bk01# du -shx *
891M ns
11G Platzhalter.iso
root@bk01:/mnt/datastore/local_backupstorage_bk01#

 

[waltar]

A placeholder.iso ... ?! You still need free space to delete it when zfs is written until end ...

 

[mhert]

really?

That is my life insurance... at least on windows servers....

Deletion is no problem on NTFS even there are 0B left...


FYI: I just checked it... deletion by shell was not a problem...

 

[mhert]

But how could this be? The quota was provable at 14.4T?!

 

[waltar]

You need more space and not really a solution to work with the last GB's of a filesystem but anyway zfs has a way for:
cat /sys/module/zfs/parameters/spa_slop_shift # maybe it's 5 or 3
echo 2 > /sys/module/zfs/parameters/spa_slop_shift
Remove eg. unneeded Platzhalter.iso file in zfs pool AND then set back the zfs parameter to orig value !!

 

[mhert]

I wanted to do as you proposed but then I realized that my deletion becomes visible in webif and now gc runs!!




Note to self: Rename Platzhalter.iso to life.saver... ;-)

 

[mhert]

But the most important question remains open:

Why quota did not work?

 

[waltar]

Maybe you write as root and that zfs quota doesn't apply to root ...

 

[mhert]

That could be right, but it is a pbs storage-target (to the cluster members), is data written as root?
And 2nd: If I set quota in pbs shell (webif); is quota set for root?

root@bk01:~# zfs set quota=14.5T local_backupstorage_bk01

 

[mhert]

I think I found the answers myself:

root@bk01:~# ls -la /mnt/datastore/local_backupstorage_bk01/
total 8267
drwxr-xr-x     4 backup backup     6 Oct  6 13:45 .
drwxr-xr-x     4 root   root    4096 Nov 20  2024 ..
drwxr-x--- 65538 backup backup 65538 Mar  2  2025 .chunks
-rw-r--r--     1 backup backup   343 Sep 27 00:00 .gc-status
-rw-r--r--     1 backup backup     0 Aug  6  2024 .lock
drwxr-xr-x     3 backup backup     3 Aug  6  2024 ns

But how to set quota for all users of a volume?

 

[MarkusKo]

Instead of quotas you could also use reservation at pool level.

zfs get reservation <pool>
zfs set reservation=<size> <pool>

If you run out of space you could remove the reservation, delete some data and then add the reservation again.
https://docs.oracle.com/cd/E19253-01/819-5461/gbdbb/index.html

 

[mhert]

so quota is user related and a reservation is for a pool (i.e for all users)?

 

[MarkusKo]

I think reservation is on zfs pool level and quota is on zfs dataset / zvol level

If your pool has multiple datasets and your quota is set too high the pool could fill up anyway so i think you have to add a reservation to the pool itself to prevent that.

*edit*
https://ikrima.dev/dev-notes/homelab/zfs-for-dummies/
had to lookup that myself

 

[waltar]

Yeah ... but pbs user is "backup" and SHOULD accept the quota set ... what's going on with zfs there ?!?

 

[mhert]

so maybe it is not MY fault at all? :-D

pbs version is 3.4.6 ...

 

[mhert]

I just wanted to add a reservation (after removing the quota) but i got this error:

root@bk01:~# zfs set reservation=15.25T local_backupstorage_bk01
cannot set property for 'local_backupstorage_bk01': size is greater than available space

root@bk01:~# zfs get available local_backupstorage_bk01
NAME PROPERTY VALUE SOURCE
local_backupstorage_bk01 available 1.93T -

root@bk01:~# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
local_backupstorage_bk01 14.5T 12.5T 2.05T - - 45% 85% 1.00x ONLINE -

Why ist the SIZE only 14.5G and the FREE 12.5T?


Did I misunderstand something?