Garbage Collection

[alanspa]

Good morning everyone
Can you please explain the logic of garbage collection?

I don't understand why on a backup repository, launching the manual prune and deleting a two week old backup, subsequently launching the garbage leaves it in "pending removal" without deleting it.

I knew that backups that are older than 24 hours are deleted immediately when I perform garbage collection.

Thank you

 

[Dunuin]

You need to wait 24 hours and 5 minutes between the prune and GC. Otherwise the GC could destroy your backups when using a filesystem with "relatime" enabled as the GC is touching chunks and marking them by updating the atime to decide what chunks to delete or keep. And with relatime enabled the atime is only updated once every 24 hours. So the time between prune and GC needs to be bigger than that.

 

[fabian]

https://pbs.proxmox.com/docs/backup-client.html#garbage-collection contains the details. chunks end up in pending removal if they are no longer referenced, but their timestamp is within 24h+5 minutes, or, after the last backup writer that is still running has been started if that was longer ago. the timestamp is updated whenever the chunk is written (so each new backup snapshot referencing it will update the timestamp), and in phase1 of the GC for chunks which are still referenced.

in practice this means if you do GC, followed by prune, followed by GC, then the first GC will have touched the chunk, and the second GC needs to be at least 24h after the first to pick up the removal.

 

[alanspa]

However, this perhaps does not explain this behavior:

I manually perform a backup weekly (this is an additional backup in addition to the "official" ones) on a dedicated storage.

Space on this storage is limited and allows me to only have two backups per server filling it to 90% capacity.

Every week I must therefore, before running the new backup, launch a prune leaving only one backup per server and then immediately run a garbage collector.
Garbage immediately deletes the backups flagged for deletion allowing me to make my own backups.
So far so good.

This week I done the backup on this storage on Monday 8 January (2 days ago) and everything went as usual, without problems.

This morning January 10th I wanted to redo this backup without waiting a week. I ran the prune leaving only one backup but this time the GC did not free up the space because it sees 2 TB in pending removals.

How come?

Thank you and goodbye

 

[Dunuin]

launch a prune leaving only one backup per server and then immediately run a garbage collector.

Correct would be: Do backup. Prune. Wait 24 hours + 5 minutes. Do GC. Ready for next backup. Do it all over again.

 

[fabian]

This morning January 10th I wanted to redo this backup without waiting a week. I ran the prune leaving only one backup but this time the GC did not free up the space because it sees 2 TB in pending removals.

How come?

something (do you have verify or sync jobs as well?) touched the chunks in the last 24h, or a different backup was running for more than 24h causing the threshold to be moved further into the past.

 

[deepcloud]

https://pbs.proxmox.com/docs/backup-client.html#garbage-collection contains the details. chunks end up in pending removal if they are no longer referenced, but their timestamp is within 24h+5 minutes, or, after the last backup writer that is still running has been started if that was longer ago. the timestamp is updated whenever the chunk is written (so each new backup snapshot referencing it will update the timestamp), and in phase1 of the GC for chunks which are still referenced.

in practice this means if you do GC, followed by prune, followed by GC, then the first GC will have touched the chunk, and the second GC needs to be at least 24h after the first to pick up the removal.

So does that mean that a GC interval below 1 day + 5min is meaningless, if so why have the option below this interval in the GUI
or i have understood this all wrong

 

[Chris]

So does that mean that a GC interval below 1 day + 5min is meaningless, if so why have the option below this interval in the GUI
or i have understood this all wrong

No, this is not useless, as this depends when you prune the backups. If you prune the backups, followed by a garbage collection, the not indexed chunks will be pending for removal until the atime limit is exceeded, the next GC will then remove them. So it could make sense for some usecases.

However, running garbage collection is very IO intensive, as it will have to read all index files for all backup snapshots and touch all the chunks still referenced. So running this with a high frequency might not be what you normally want. But I seen no reason to force a limited schedule.

 

[deepcloud]

So lets assume that you want a backup policy like

Keep last 6 hourly backups
Keep last 7 days, 4 Weeks, 12 Months and 3 years.
What would be the best GC and Prune settings for this

 

[Dunuin]

Depends on how much your data changes. With only minimal changes it for example wouldn't hurt to only run that GC once per day or week even if that would mean there are 18 or 162 hourly backups waiting for removal.