PBS datastore 100% full - manual GC/prune does not work

[zaphyre]

Hi, I have two PBS servers in my setup.
The primary one is ok and working as expected.

The second PBS has a sync job to basically clone the content from the first PBS. Thats all it does. Well, apparently I didnt understood that te datastore on the second pbs needed its own GC/Prune job config - I thought of it more of a replication - "mirroring" only the content from the 1st PBS which has the respective GC/prune config.

Seems I have misse --remove-vanished <boolean> (default=false) option in the sync job.

Result is the second datastore is now at a 100% full, with no space left one device.

Now, when I create a prune task with the GUI it fails due to no space left on device.
Is there a way on the cli or gui to manually remove some backups to get some free space?

Thanks for help!

 

[Matthias.]

If you do not care about the data on the second one, you can delete the files in the .chunks folder.
This will permanently damage all backups using that chunk. Of course, if you can just resync from the first PBS, this might not matter.

Using proxmox-backup-debug you could figure out which backups use what chunks.

All in all, it is a rather dangerous thing to do to your backups, please keep that in mind.

I thought of it more of a replication - "mirroring" only the content from the 1st PBS which has the respective GC/prune config.

If you check the "remove vanished" option, after syncing the second PBS will delete all local backups that are not on the remote (first) pbs.

 

[zaphyre]

If you do not care about the data on the second one, you can delete the files in the .chunks folder.
This will permanently damage all backups using that chunk. Of course, if you can just resync from the first PBS, this might not matter.

Using proxmox-backup-debug you could figure out which backups use what chunks.

All in all, it is a rather dangerous thing to do to your backups, please keep that in mind.


If you check the "remove vanished" option, after syncing the second PBS will delete all local backups that are not on the remote (first) pbs.

Hi Matthias and thanks for your quick help!

As the first PBS still looks fine - I would delete the .chunks folders content, then set the vanish option on the sync job (that I totally missed) and finally do a resync or wait for the next sheduled run.

So if the the vanish option is set I would not need GC/prune setting on the second PBS, right?

 

[Matthias.]

Actually, don't delete from .chunks. It might break the system
Safer and better is to delete some snapshots from the datastore, that is, the folders in <path to datastore>/[ct|vm]/<vmid>/ named with date and time of the backup, e.g. 2022-04.22T12:01:00Z, then garbage collect.

Yes, with vanish you dont need GC or prune. But please note that with vanish, if you accidentally delete all backups on the first PBS and sync, all backups on the second one will be deleted aswell.

 

[zaphyre]

Actually, don't delete from .chunks. It might break the system
Safer and better is to delete some snapshots from the datastore, that is, the folders in <path to datastore>/[ct|vm]/<vmid>/ named with date and time of the backup, e.g. 2022-04.22T12:01:00Z, then garbage collect.

Okay, will do this way.

 

[zaphyre]

Yes, with vanish you dont need GC or prune. But please note that with vanish, if you accidentally delete all backups on the first PBS and sync, all backups on the second one will be deleted aswell.

Okay, yeah, makes definitely a lot of sense to have vanishing off by default. So I just thought about mirroring the two PBS' which on would achieve with the vanish option. But normaly I can achieve different retention shedules if needed with leaving vanishing off and implementing a different gc/prune on the second PBS. Now, everything makes sense

Trying to get my second PBS responsive again and then think about my DR concept/plan.

 

[fabian]

to get it responsive again the following should work:
- disable any incoming backups/sync jobs
- remove a few snapshot directories manually (these only contain metadata, this is effectively the same as pruning them - you can either remove some that are already gone on the source side to mimic remove_vanished, or remove the last N in each group, these will then be re-synced if they still exist on the source. or you can remove some that you don't care about )
- attempt to run GC (it needs a bit of space for atime updating)
- if GC fails for lack of space, remove some more snapshot dirs
- attempt GC again, rinse-repeat until GC works
- re-enable sync/backups/..