prune removed more than expected

[james-01]

PVE node: 7.4-17
PBS: 3.1-2

PBS built just over a month ago and yesterday started to configure PVE backup jobs with retention settings:
keep-last 4
keep-weekly 52
keep-monthly 24
keep-yearly 10

1st PVE backup job with the retention settings resulted in removal of the only backup (in PBS) from September. I was expecting the September backup to be retained and had tried out settings first using the prune simulator. (This removal occurred with several VM backups)

Note there are no prune jobs configured in PBS - only the retention settings for the PVE backup job.

Example

task log for prune (from PBS task history):

2023-12-29T00:10:28+00:00: retention options: --keep-last 4 --keep-weekly 52 --keep-monthly 24 --keep-yearly 10
2023-12-29T00:10:28+00:00: Starting prune on datastore 'store1', root namespace group "vm/101"
2023-12-29T00:10:28+00:00: vm/101/2023-11-29T09:25:41Z remove
2023-12-29T00:10:28+00:00: vm/101/2023-12-01T19:44:27Z keep
2023-12-29T00:10:28+00:00: vm/101/2023-12-04T02:10:01Z keep
2023-12-29T00:10:28+00:00: vm/101/2023-12-12T22:49:23Z protected
2023-12-29T00:10:28+00:00: vm/101/2023-12-13T18:17:29Z protected
2023-12-29T00:10:28+00:00: vm/101/2023-12-18T00:10:02Z keep
2023-12-29T00:10:28+00:00: vm/101/2023-12-25T00:10:04Z keep
2023-12-29T00:10:28+00:00: vm/101/2023-12-29T00:10:01Z keep
2023-12-29T00:10:28+00:00: TASK OK

list of backups in the PBS datastore after the prune (to show no backups from September retained)

root@crusader:~# pvesm list banshee-store1 --vmid 101
Volid                                             Format  Type             Size VMID
banshee-store1:backup/vm/101/2023-12-01T19:44:27Z pbs-vm  backup    21475378379 101
banshee-store1:backup/vm/101/2023-12-04T02:10:01Z pbs-vm  backup    21475378377 101
banshee-store1:backup/vm/101/2023-12-12T22:49:23Z pbs-vm  backup    21475378353 101
banshee-store1:backup/vm/101/2023-12-13T18:17:29Z pbs-vm  backup    21475378351 101
banshee-store1:backup/vm/101/2023-12-18T00:10:02Z pbs-vm  backup    21475378376 101
banshee-store1:backup/vm/101/2023-12-25T00:10:04Z pbs-vm  backup    21475378377 101
banshee-store1:backup/vm/101/2023-12-29T00:10:01Z pbs-vm  backup    21475378268 101
root@crusader:~#

Any ideas on what I have got wrong here?

Thanks
James

 

[sterzy]

Any ideas on what I have got wrong here?

I think you mean November not September? At least I don't see a backup from September anywhere in your provided logs. Anyway, if you are wondering why the backup "vm/101/2023-11-29T09:25:41Z" was removed, the answer is simple: the 29.11. is in the same week as the 01.12. so from that week PBS will only keep one backup "vm/101/2023-12-01T19:44:27Z". You don't have enough backups for keep-monthly to kick in yet.

The options you chose tell PBS the following:

• keep-last = 4: keep the last 4 backups, after you've determined which backups that applies to proceed with,
• keep-weekly = 52: keep the last backup from the remaining 52 weeks, remove all other backups from the same week, afterwards
• keep-monthly = 24: keep the last backup of the 24 months starting after all other rules
• keep-yearly = 10: keep the last backup of the 10 years starting after all other rules

Think of it as PBS traversing the list of all backups once. It first marks the first four backups it finds as "keep due to keep-last = 4". It then looks at the remainder of the list and determines the last backup of each remaining week that has at least one backup. It chooses at most 52 of those. It marks all other backups in that time period as "to be removed". Then it looks at the remainder of the list, and keeps at most the 24 last backups in the remaining months. All other backups in that 24 months are marked to be removed. Afterward it will look at a time period of ten years starting from the last kept backup and keep the last backup of each of thos ten years.

So at the point where you get to your monthly backups, PBS has already run out of backups. The option keep-monthly will only start to kick in once there are at least four backups and at least 1 backup for each of the preceding 52 weeks.

You can also see this in the prune simulator if you enter your settings: https://pbs.proxmox.com/docs/prune-simulator/

 

[james-01]

Thanks Stefan
I see where my slight misunderstanding was now. (Yes I did accidentally refer to September when I should have written November!)

regards
James

 

[sterzy]

I see where my slight misunderstanding was now. (Yes I did accidentally refer to September when I should have written November!)

No problem, please mark this thread as solved by clicking "Edit Thread" above and selecting the "Solved" prefix if you don't have any further questions.