PBS Prune and GC vs Job / Storage retention

[ozdjh]

Hi

I understand that a PVE backup job without retention config inherits the config of the PVE storage that's being use for the backup. But what's the purpose of Prune and GC on the PBS target itself? Does that get applied if the job has no retention config and the PVE storage has no retention config either?

I'm guessing that we just setup our Daily/Weekly/Monthly retention on the PVE backup job and tell PBS to keep whatever it's sent. But that feels like I'm misunderstanding something (which I probably am). If someone can share some clue with me I'd be grateful.

Thanks
David

 

[fiona]

Hi,
prune on PBS will run completely independently. If you have retention settings on both PVE and PBS, both will be applied. Note that you can also backup to PBS from a different source than PVE via proxmox-backup-client. And garbage collection is for cleaning up chunks that are not referenced by any backup snapshot anymore, so that's also independent.

 

[ozdjh]

Ok, thanks Fabian. So if we are only backing up from PVE and we are using a per-job retention config, we don't need to rely on PBS pruning at all, and GC should be run every now and then just in case something's left unreferenced?

 

[fiona]

Ok, thanks Fabian. So if we are only backing up from PVE and we are using a per-job retention config, we don't need to rely on PBS pruning at all, and GC should be run every now and then just in case something's left unreferenced?

Yes. But I wouldn't say "just in case" , because pruning will not free up any chunks itself (it just removes metadata). If you have a lot of changing data in the backups, you might want to run GC more often accordingly.

 

[ozdjh]

Oh, so if you delete a backup from the PVE storage it doesn't delete the chunks even if that was the last reference to the chunks? I assumed the chunks would be reclaimed as soon as nothing was actually using them. Are you saying that running the GC is the only way to reclaim that space?

 

[fiona]

Oh, so if you delete a backup from the PVE storage it doesn't delete the chunks even if that was the last reference to the chunks? I assumed the chunks would be reclaimed as soon as nothing was actually using them. Are you saying that running the GC is the only way to reclaim that space?

No, that's what garbage collection is for. Otherwise it would require collecting metadata for each chuck indicating who still references it, and that is very easy to get wrong and can become outdated in failure scenarios. GC does it the other way around and iterates the backup snapshots to see if chunks are still needed. And yes, it's the only way to reclaim (non-metadata) space.

 

[ozdjh]

Ok, understood. Thanks for the clarification. We may run it more frequently then

 

[mindio]

Hi All,

I am a bit confused, so would like revisit this topic again.
So if I have a single backup task in PVE and happy with all retention rules, there is no need to repeat the same prune settings on PBS side, but is advised to have GC rule. Is that enough to be a good practice for prune and GC purpose?

 

[fiona]

Hi,

Hi All,

I am a bit confused, so would like revisit this topic again.
So if I have a single backup task in PVE and happy with all retention rules, there is no need to repeat the same prune settings on PBS side, but is advised to have GC rule. Is that enough to be a good practice for prune and GC purpose?

yes, there is no need to repeat the prune settings. And yes, you still need garbage collection in PBS. Because in PBS, a backup is essentially just metadata files referencing a collection of data chunks. When a backup is removed, only the metadata files are removed. Garbage collection is what checks if any chunks are not referenced by any backup and can actually free up space.

If you are planning to add more backups with the same retention policy, it's likely better to configure the retention on the PBS side instead. If you are planning to add more bacukps with different retention policies, it's easier to keep it at the client side.