[solved] configuring prune & gc

[Elleni]

We do monday to friday 9 backups à day. I want to keep all backups for a week, then one per day for the next three weeks before, then one monthly for a year , and one yearly for 10 years.

Thus I configured:
hourly 45
day 5
week 4
month 12
year 10

I would have expected that after pruning & gc it keeps last week's 45 hourly backups and then only one per day, but it does not prune anything and keeps all backups. What am I doing/understanding wrongly?

 

[che]

We do monday to friday 9 backups à day. I want to keep all backups for a week, then one per day for the next three weeks before, then one monthly for a year , and one yearly for 10 years.

Thus I configured:
hourly 45
day 5
week 4
month 12
year 10

I would have expected that after pruning & gc it keeps last week's 45 hourly backups and then only one per day, but it does not prune anything and keeps all backups. What am I doing/understanding wrongly?

Hi,
there is a prune simulator which helps you set up your schedule correctly, see https://pbs.proxmox.com/docs/prune-simulator/

 

[Elleni]

I have seen this, and thus configured as described according to what I have seen in the simulator. Nevertheless I see that if I go back one week, there are still kept 9 backups a day and I don't understand why.

 

[fiona]

Hi,
when is the prune job scheduled? Could you share a log of the latest prune task? What is the output of

proxmox-backup-client prune vm/<ID> --repository <name> --keep-last=45 --keep-daily=5 --keep-weekly=4 --keep-monthly=12 --keep-yearly=10 --dry-run

for some backup group with lots of backups?

 

[Elleni]

I did not schedule it yet, as I just run it manually for testing.

Following the Log from the GC job which did not remove anything:
https://pastebin.com/SkAhNRf1

Dry run:
https://pastebin.com/Lx4huDTW

The dry run indicates to do as expected. I'll try to run gc again from gui and report back.

I have to add, that I did the prune testing not on the pbs-backup repository where the backups are taken, but on a second repo named sandisk where I synced the backups to. I used the local remote configuration as workaround to be able to sync to a second removable disk.

 

[Elleni]

Tried again, but it does not prune anything away:

2020-12-09T11:47:53+01:00: Removed garbage: 0 B
2020-12-09T11:47:53+01:00: Removed chunks: 0
2020-12-09T11:47:53+01:00: Original data usage: 178.84 TiB
2020-12-09T11:47:53+01:00: On-Disk usage: 1.22 TiB (0.68%)
2020-12-09T11:47:53+01:00: On-Disk chunks: 614862
2020-12-09T11:47:53+01:00: Deduplication factor: 146.30
2020-12-09T11:47:53+01:00: Average chunk size: 2.08 MiB
2020-12-09T11:47:53+01:00: TASK OK

gui configuration:
https://paste.pics/925051b976dd4bc72b3895dfec3681e2

 

[fiona]

Garbage collection and pruning are two different types of jobs. I don't think you can start a datastore wide prune manually yet, only for a specific backup group. That is, using the CLI command without the dry-run option or the scissor icon in the GUI.

If you don't have any prune tasks in the log, it means that no prune was executed yet, so there is nothing to garbage collect either.

The dry-run command showed you what a prune would do, so it should be safe to enable scheduled prunes. And 24 hours (because of some technicalities this is necessary) later garbage collect will clean up the unneeded chunks.

 

[Elleni]

Ah, ok I see. So I scheduled the prune task every half an hour in order to trigger a pruning job, and will then see if it works as intended, to finally set the prune job accordingly. Thanks for your clarification. I'll report back if it worked.

 

[Elleni]

Looking good.

2020-12-09T13:33:30+01:00: Removed garbage: 0 B
2020-12-09T13:33:30+01:00: Removed chunks: 0
2020-12-09T13:33:30+01:00: Pending removals: 499.68 GiB (in 277424 chunks)
2020-12-09T13:33:30+01:00: Original data usage: 54.76 TiB
2020-12-09T13:33:30+01:00: On-Disk usage: 752.03 GiB (1.34%)
2020-12-09T13:33:30+01:00: On-Disk chunks: 337438
2020-12-09T13:33:30+01:00: Deduplication factor: 74.56
2020-12-09T13:33:30+01:00: Average chunk size: 2.28 MiB
2020-12-09T13:33:30+01:00: TASK OK

Thanks a lot for helping me understand that the start GC does not imply a prune job.