[SOLVED] Backup retention settings

[ahwang]

There appears to be mulitple places where one can configure retention.

On PBS, under the datastore->prune&GC, it looks like you can specify retention. When you configure a new storage on PVE using a PBS, it looks like you can also specify retention on the storage. And finally, when you configure a backup job in PVE, it also looks like yet another place to configure retention. This makes a total of 3 different places that I've seen involving backups/storage where you can specify retention.

So it's a bit confusing where one should configure the retention.

Also, if retention were configured in multiple places, is there some sort of hierarchical precedence? Or does the smallest retention value take effect (ie. keep 30 dailies on PBS, and keep 7 dailies on backup job would result in keeping only 7 days) since each respective setting is applying the retentions they have resulting on the smallest retention taking effect?

Also, on the PVE side of things both storage and backup job have a checkbox for "keep all backups". What exactly does this do? If you tick this, does it override all retentions and nothing gets aged out? Is this like a master switch to prevent any of the backups from ever getting purged regardless of the retention settings?

Can someone explain the differences in these retention settings and what is best practices? Is everyone just configuring retention in one place to avoid confusion?

 

[Dunuin]

Also, if retention were configured in multiple places, is there some sort of hierarchical precedence?

Yes. The retention on PBS and the PVE backup job will always take effect. So both run in parallel and the more aggressive one wins as both will prune but the more aggressive one will prune what the less aggressive one had left over.
I remember a staff member explaining that the retention of the storage in PVE is secondary and will be used as a fallback (but can'T remember details).

I always ignore all retentions on the PVE side and only set them for the datastore/namespace in PBS. Then I also set up PBS users without the privilege to prune/destroy backups. So in case my PVE host will be hit by ransomware or similar the compromised PVE won'T be able to destroy the backups on the PBS.

Also, on the PVE side of things both storage and backup job have a checkbox for "keep all backups". What exactly does this do?

That means the backup job on PVE won't prune anything. Not sure, but I think the PBS will still be pruning in case a retention is set for the datastore.

 

[ahwang]

I was kind of leaning towards that approach as well. Easier to manage retention from a single place and on the PBS instead of on individual PVE jobs. I suppose the retention on the PVE jobs gives you the flexibility if you really need to customize retention on a backup by backup basis. But that isn't the case for me and I'd much rather keep it simple and have a single retention setting and view.

So in summary, I think what I will do is just tick all the "keep all backups" on the client side (ie. PVE storage and backup jobs) to prevent them from purging at all and just set the retention centrally on the PBS datastore.

I like your idea of PBS user without the capability to prune/destroy backups in the event of malware and/or attack. Hadn't thought of this.

 

[Dunuin]

So in summary, I think what I will do is just tick all the "keep all backups" on the client side (ie. PVE storage and backup jobs) to prevent them from purging at all and just set the retention centrally on the PBS datastore.

Yeah, thats how I do it and happy with it.

I like your idea of PBS user without the capability to prune/destroy backups in the event of malware and/or attack. Hadn't thought of this.

Also have a look at the Ransomware Protection section in the manual: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

 

[SilkBC]

If you want to create different "tiers" of retention, though, I think you would have to configure this on the "client" (or Datacenter) side? So for example, you have some VMs that you require something like the last 7 daily+last 4 weekely+last 12 monthly+last 2 yearly, you would create that backup job and have to set the retention in there.

But if you have another group of VMs that are not as critical, so you have them in a separate backup job that only retains the last 7 daily backups.

And so on. I don't see an option to do that per-Job in PBS; only in the Datacenter side of things in PVE?

 

[niteshadow]

What I do is create different namespaces on PBS (primary, secondary, etc.)
Then on PVE I have different tiers of backup jobs/schedules to namespace. No need to prune on PVE.
Back on PBS add a prune job/schedule for each namespace.

 

[SilkBC]

What I do is create different namespaces on PBS (primary, secondary, etc.)
Then on PVE I have different tiers of backup jobs/schedules to namespace. No need to prune on PVE.
Back on PBS add a prune job/schedule for each namespace.

OK, so it looks like the Namespace is specified when adding the Proxmox Backup Server to storage, but you can only specify one namespace, so if one were to have different tiers of backup (for different retention), then the same Proxmox Backup Server would need to be added multiple times to PVE (once per namespace)?

 

[niteshadow]

Correct

 

[Dunuin]

Yes, but one thing to keep in mind:
Once you backup the same VM to two different namespaces this will drop the dirty-bitmap and VM backups will be slower as then then whole virtual disk will have to be read and hashed each time and not only the parts of the virtual disks that changed since the last backup.
To prevent that you could always backup those VMs to the same namespace and then work with local sync jobs on the PBS to copy those backup snapshots to different namespaces.

 

[mow]

BTW, is there an option to manually "pin" a certain backup? E.g. you have a VM you don't need for a while, so you backup it to the pbs and remove it from the pve but want to be able to restore it eventually?

 

[Dunuin]

You can enable protection for each backup snapshot so you can't delete it by accident and it won't be pruned until you disable the protection again. But be aware that a sync job will drop that protection flag and it then might still be pruned on the pulling PBS.