Resize ZFS pool after running out of space (PBS datastore full)

[mhert]

Over the weekend my pbs datastore is running out of space and now I'm unable to run a garbage collection.

This is the output of discus:



and this is the output of zpool list:



What is the meaning of FREE and second question: How could this help me out of my dilema? Is is possible to use the free space or resize the pool in a way which make a garbage collection possible?

Regards,

itiser

 

[mhert]

Nobody? :-(

 

[Johannes S]

For me it means exactly what is says: You have used 14.4 TB of 14.5 TB available disk space and thus can't continue (the backup user used by PBS can't use the last inch of the volume because that's left free for the root user). How did you build your ZFS pool? Is it a RAIDZ or a mirror? In any case you should be able to add further disks to reszize the pool to have more space, but depending on the pool layout your options are different.

Another option might be add another larger datastore (e.G. via NFS/CIFS or an external hard drive) sync everything on the existing datastore to it with a local sync job, run garbage collection on the temporary datastore. Afterwards destroy your old datastore and resync everything from the temporary datastore like proposed by Dunuin in this thread:

Post in thread 'PBS Datastore 100% full'

Oct 30, 2023

Garbage collection because no space left, can I deleted them manually? Or any other thing I can do?

You don't know which chunk belongs to what backup snapshot. So by randomly deleting chunks you will probably corrupt most of your backups snapshots.

Then there aren't many options if you can't run a GC.

If you don't care about losing all your backups you could just delete the datastore folder and start from scratch with a new datastore.

If you care about your backups you can only:
1.) buy more disks and add them to the pool so you get more space
2.) mount a bigger filesystem...

• Dunuin

• 

Another alternative would be to use zfs send/receive (if your current datastore and your temporary location have zfs) or more hacky approaches like this one (be warned: There is a high chance by loosing data if you do something wrong):

Post in thread 'How to recover from 100% disk use'

Feb 22, 2024

You don't really need to do that. I don't remember the exact steps, but you can try:

1.- Place datastore in maintenance, read only
2.- Move some files from the .chunks directory to some other drive. Having like 200MB free should be enough.
3.- Run GC and hope it's able to delete some chunks. You may need to wait 24h+5m so the GC is really able to delete something [1]. You will get errors about missing chunks (those you moved in step 2).
4.- Move back the chunks to the same directory you moved them from.
5.- Run a full verify.
6.- Run GC again.
7.- If enough free space, remove maintenance...

• VictorSTS

• 

Or this one (another more hacks again think twice before executing a command, only do it if you are sure about the possible consequences):

E

Post in thread 'I can´t get space if i delete backups on PBS'

Apr 18, 2024

is there not a way to force it?

There is a non-recommended way to delete the data immediately ...

find /path2pbs-datastore/.chunks -type f -print0 | xargs -0 touch -d "-2 days"

.... and GarbageCollect.

• Ernst T.

• 

 

[news]

Run zpool trim <pool> and set zfs set quota=<size> <pool> after you clean pbs space.

 

[Johannes S]

Yes setting a quota to something like 80% of the pool size is also a good idea because then you have some breathing room if the quota limit is reached: You can then temporary remove the quota, remove data as needed and add it again.