PBS prune is removing backups that should be kept

[VictorSTS]

I have a PVE cluster (v8.1.4) doing backups to a PBS server. The backups are done every 3 hours with a */3:11 schedule.
On PBS (v3.2-2), I have a prune policy setting like this:

        keep-daily 30
        keep-hourly 24
        keep-last 3
        keep-monthly 12
        keep-weekly 4
        keep-yearly 2

IIUC, this should keep a copy of the last day of the month for 12 months and for the last day of the year. If you configure this parameters in prune simulator [1] it does confirm this.

PBS is removing that last backup of the last day of the month of all VMs. Focusing on the backup of VM 1008, I looked in logs to find when the backup of 2025-02-28 was removed. That happened on 2025-03-04 "remove vm/1008/2025-02-28T23:11:02Z":

/var/log/proxmox-backup/tasks# find . -type f -name "*prune*" -exec grep -H 'remove vm/1008/2025-02-28T' {} \;

./F0/UPID:PBS01:00000AFC:000007F0:00005384:67C75BC0:prunejob:DS01:root@pam::2025-03-04T21:00:00+01:00: remove vm/1008/2025-02-28T11:11:03Z
./F0/UPID:PBS01:00000AFC:000007F0:00005384:67C75BC0:prunejob:DS01:root@pam::2025-03-04T21:00:00+01:00: remove vm/1008/2025-02-28T14:11:01Z
./F0/UPID:PBS01:00000AFC:000007F0:00005384:67C75BC0:prunejob:DS01:root@pam::2025-03-04T21:00:00+01:00: remove vm/1008/2025-02-28T17:11:02Z
./F0/UPID:PBS01:00000AFC:000007F0:00005384:67C75BC0:prunejob:DS01:root@pam::2025-03-04T21:00:00+01:00: remove vm/1008/2025-02-28T23:11:02Z
./F0/UPID:PBS01:00000AFC:000007F0:00004CBB:67C60A40:prunejob:DS01:root@pam::2025-03-03T21:00:00+01:00: remove vm/1008/2025-02-28T02:11:02Z
./F0/UPID:PBS01:00000AFC:000007F0:00004CBB:67C60A40:prunejob:DS01:root@pam::2025-03-03T21:00:00+01:00: remove vm/1008/2025-02-28T05:11:02Z
./F0/UPID:PBS01:00000AFC:000007F0:00004CBB:67C60A40:prunejob:DS01:root@pam::2025-03-03T21:00:00+01:00: remove vm/1008/2025-02-28T08:11:02Z
./F0/UPID:PBS01:00000AFC:000007F0:00011ED0:67EEDAB0:prunejob:DS01:root@pam::2025-04-03T21:00:00+02:00: remove vm/1008/2025-02-28T20:11:03Z

I've checked other PBS servers and this seems to affect other versions too: the last backup of each month isn't always the last backup of the last day of the month. This seems to work ok for daily and weekly retentions, were the last backup of each period is kept correctly. There are no prune policies in PVE and all PVE reach PBS with a user that only has Datastore.Audit and Datastore.Backup, so there is no chance that PVE can prune backups.

Why is it removing those backups? I really need RPO predictability, specially in long term backups.

[1] https://pbs.proxmox.com/docs/prune-simulator/index.html

 

[SteveITS]

Are you ending up with no backup from February? What happened in March?

I might start by updating to 3.4 and testing again…

 

[VictorSTS]

@SteveITS Same behavior with v3.1-x to v3.4-x. Didn't notice before because we seldom need to recover anything a few days old, fortunately.

I've checked other PBS servers and this seems to affect other versions too:

Are you ending up with no backup from February? What happened in March?

Thanks for pointing that out. February's backup that got kept was on vm/1008/2025-02-23T20:11:01Z. Same situation in every month: the backup of the last day isn't kept, but instead the backup of the last sunday of the month.

At the very least, there is a discordance between what the prune simulator says and what actually happens.

 

[SteveITS]

Hmmm, if I plug numbers in to the simulator it shows keeping the Sunday April 27 backup:
(edit: which is what it does, for me)


I would read it as, "keep a weekly backup" so it keeps Sundays, and then later when it comes time to prune that, it considers "the last one of the month is the 27th, keep that."

 

[Chris]

Hi,

IIUC, this should keep a copy of the last day of the month for 12 months and for the last day of the year.

please note: This parameter does not mean what you intend it to. As also stated in the prune simulator: Keep backups for the last <N> months. If there is more than one backup for a single month, only the latest is kept. So if the backup snapshot for the last day of a month was already pruned by e.g. the weekly setting, the monthly backup will be a different day (not the last day of the month). So this also depends on when the prune job runs, and what the actual datastore contents are at given time.

You could remove the keep weekly and increase the keep daily to 31 to have the last day of the month always present.

 

[VictorSTS]

Look for the backups kept due to keep-monthly, in blue (you may need to increase Duration). Those green ones are correctly kept due to the keep-weekly policy, hence the last backup on Sunday is kept.

I would read it as, "keep a weekly backup" so it keeps Sundays, and then later when it comes time to prune that, it considers "the last one of the month is the 27th, keep that."

That's what is happening because PBS prunes the last backup of each month. When time comes to keep the last backup of each month, it uses that of the last Sunday, as it's the last of that period.

So if the backup snapshot for the last day of a month was already pruned by e.g. the weekly setting, the monthly backup will be a different day (not the last day of the month)

I think this could be improved. Prune knows I want 12 monthly backups and should keep the last backup of the month, independently of what other prune policies are. Same for keep-yearly.

Also, prune simulator is misleading as it does not reflect this behavior: it always keeps the last day of the month, even when using keep-weekly settings.

You could remove the keep weekly and increase the keep daily to 31 to have the last day of the month always present.

Or add a backup hook script that adds the protected bit on given dates, as this would allow me to keep those weekly backups.

 

[SteveITS]

prune simulator is misleading as it does not reflect this behavior: it always keeps the last day of the month, even when using keep-weekly settings

Ah, I see what you mean:

 

[Chris]

Also, prune simulator is misleading as it does not reflect this behavior: it always keeps the last day of the month, even when using keep-weekly settings.

Yes, but that is if all of the given snapshots would be present at this exact point in time when the prune job is executed. If you, e.g. change the keep weekly, you will see that this might influence the keep monthly. For some configurations the kept monthly snapshot will not be on the last day of the month, even if all the snapshots would be present at given time. Please see also https://forum.proxmox.com/threads/prune-simulator-issue.162664/post-751076

There has been a request for time based retention settings, see https://bugzilla.proxmox.com/show_bug.cgi?id=6074

 

[Chris]

Or add a backup hook script that adds the protected bit on given dates, as this would allow me to keep those weekly backups.

Yes, that is also possible, might even be better for your use case as no other job or user interaction can then prune that snapshot unless explicitly removing the protected flag. Edit: would however require to remove it at some point if you want to prune it.