PBS Prune Simulator - monthly backups

[thebream]

Have installed Proxmox Backup Server, and has been working great for a few weeks now.

Trying to understand how pruning works, and ran the simulator at:
https://pbs.proxmox.com/docs/prune-simulator/

Used these parameters, similar to what I am using for real:
Prune options
last: 2
daily: 6
weekly: 4
monthly: 6

Simulated backup schedule
day of week: Mo, Tu, We, Th, Fr, Sa, Su
backup schedule: 02:00
number of weeks: 16

This shows the following backups having been retained:
2022-09-13 Tu last: 1
2022-09-12 Mo last: 2
2022-09-11 Su daily: 1
2022-09-10 Sa daily: 2
2022-09-09 Fr daily: 3
2022-09-08 Th daily: 4
2022-09-07 We daily: 5
2022-09-06 Tu daily: 6
2022-09-04 Su weekly: 1
2022-08-28 Su weekly: 2
2022-08-21 Su weekly: 3
2022-08-14 Su weekly: 4
2022-07-31 Su monthly: 1
2022-06-30 Th monthly: 2
2022-05-31 Tu monthly: 3

Note the monthly ones:
2022-08-31 Wed (last day of August) was not retained, as expected from the comments on the simulator page:
"The retention options are processed in the order given above. Each option only covers backups within its time period. The next option does not take care of already covered backups. It will only consider older backups.

For example, in a week covered by keep-weekly, one backup is kept while all others are removed; keep-monthly then does not consider backups from that week anymore, even if part of the week is part of an earlier month."

2022-07-31 Sun (last day of July) WAS retained, also as expected, as it was a Sunday - so was originally kept as a weekly backup.

So, my question is, why were 2022-06-30 and 2022-05-31 retained? These were both mid-week, so should have been discarded because the following Sunday backups would have been retained as weekly backups?

In my own setup, which has only been running for about 6 weeks, the 2022-08-31 backup HAS been discarded - I don't have enough history to see what has happened for the other monthly backups.

Am I misunderstanding how the monthly retention works, or is there something wrong with the simulator?

 

[Matthias.]

Sun, 31 Jul covers July. Then it goes to June, finds the most up-to-date backup there: Thu, 30 Jun.
June is covered, go to May, find latest backup, and so on.

You start from the current date and process the pruning going back in time and in the order given, i.e first latest, then hourly, up to yearly.
When pruning, you keep the most recent backup and skip to the start of the timeframe. i.e with hourly, you would skip to the previous hour, with daily to the previous day, with monthly to the previous month. then you keep the most recent backup, rinse and repeat.

keep-weekly: 4 means that a total of 4 weekly backups are retained. So once you have passed that number (2022-09-04 Su weekly: 1, 2022-08-28 Su weekly: 2, 2022-08-21 Su weekly: 3, 2022-08-14 Su weekly: 4), weekly is not relevant at all

I hope that helps and I understood your question correctly

 

[thebream]

Thanks for your response, but monthly backups seem to be slightly different. Quoting from the simulator page again:
"in a week covered by keep-weekly, one backup is kept while all others are removed; keep-monthly then does not consider backups from that week anymore, even if part of the week is part of an earlier month"

Last backup for August - 31st, a Wed, has not been retained
(last backup for June - 30th, a Sun, was covered by weekly backup)
But last backup for Jun - 30th, a Thu, HAS been retained, according to the simulator - I would expect this to NOT be retained, like Aug 31.

Given the above schedule, a prune done at noon on Thu Jul 7 should leave this:
Jul 7, 6 ("last" x 2) / Jul 5, 4, 3, 2, 1, Jun 30 ("day" x 6) / Jun 26 ("week")
Then, on Jul 8, it would look like this:
Jul 8, 7 ("last" x 2) / Jul 6, 5, 4, 3, 2, 1 ("day" x 6) / Jun 26 ("week")
[keep-monthly will not consider Jun 30 (even though it is part of an earlier month), because keep-weekly will retain Sun Jul 4]
And by Jul 11, it should look like this?
Jul 11, 10 ("last" x 2) / Jul 9, 8, 7, 6, 5, 4 ("day" x 6) / Jul 3, Jun 26 ("week")

 

[Matthias.]

Last backup for August - 31st, a Wed, has not been retained

(last backup for June - 30th, a Sun, was covered by weekly backup)
But last backup for Jun - 30th, a Thu, HAS been retained, according to the simulator - I would expect this to NOT be retained, like Aug 31.

The backup for August 31 wasn't retained because that month is already covered by the other backups. Also because there is still a keep-weekly "missing".

Jun 30 is retained because it's the latest backup in June, and no other, later backup covers June.

Given the above schedule, a prune done at noon on Thu Jul 7 should leave this:

Jul 4 2022 was a Monday, but other than that, seems right to me. Might want to check by changing your time and reloading the site.
Either way, you were wondering about monthly backups, I'm a bit confused why you only talk about the weekly ones now.

Feel free to have a look at the code yourself. You can find it in https://git.proxmox.com/?p=proxmox-...a58267a77a75e5b815dfe466da590507f372e;hb=HEAD (the actual pruning) and https://git.proxmox.com/?p=proxmox-...3461c9e7869c2a242143e4c1191cee5cda64d;hb=HEAD (the simulator), respectively.

 

[thebream]

Thanks again Matthias, let me try to put this another way.

Either way, you were wondering about monthly backups, I'm a bit confused why you only talk about the weekly ones now.

Because - "in a week covered by keep-weekly, one backup is kept while all others are removed; keep-monthly then does not consider backups from that week anymore, even if part of the week is part of an earlier month"

So the weekly backups affect what happens with monthly backups.

Might want to check by changing your time and reloading the site

That's an excellent idea! I did try this in a VM (after backing it up with PBS of course ), but trying to load the simulator page complained that "Your clock is behind".

Nevertheless,

The backup for August 31 wasn't retained because that month is already covered by the other backups

Agree - because there was a Weekly backup from Sun Aug 28. And that will eventually be retained as a Monthly backup.

So, by the same logic, the Thu Jun 30 backup should have been pruned - leaving Sun Jun 26 as the "monthly" backup:

[mid July] Keep-weekly would be keeping Sun Jul 3, Sun Jun 26 . . .​

​

And, "in a week covered by keep-weekly, one backup is kept. . ." - i.e. Jul 3.​

". . . while all others are removed" - Thu Jun 30 will be pruned when no longer covered by keep-last or keep-daily.​

​

Leaving Sun Jun 26 as the "monthly" backup for June when it is no longer covered by keep-weekly.​

But the simulator, when run now (September), is showing that the Jun 30 has been retained as the monthly backup.

 

[Matthias.]

The weekly backup covers its week, from previous monday to next sunday.
The monthly backup covers its month, from 01st to 28th/29th/30th/31st of the month