Advanced pruning issues? looking for work arounds.

[rwithd]

So we have some servers that have different intervals for backups. 30m, (internal databases) 1h, (RDP servers) 24h, (Servers that don't have much local data but still have some regular changes.

We can configure purge policy in the cluster, But that will run purge and Garbage Collection as part of every backup ok, but then i can have multiple purge and GC processes running at once. This Causes us issues because we want to keep the last 4Hours of backups from the 30m group (keep-last=8), but don't want to add 8 Days to our 24h Group. I understand that the Purge&GC configs are kept in the vzdump file (I THINK)? Can PBS use those configs.

The next issue is that We have setup a PBS in AWS, It works. But we have no way of limiting what gets sent to the remote, We don't want backups being sent to the remote every 30 min just one daily, the backups are small so its not a huge deal. But seems like a waste of bandwidth.

So I guess what i'm looking for are Per Backup/group, Retention policies, and Per Remote sync policy. are those on the road map?

P.S. I know I can create multiple data stores, with different policies. but that's a pain, I already have 4 shared storage devices per node, I don't want to add 2 more. should i just give up and do this?

 

[rwithd]

So one downside to having to use multiple Datastores would be a large Decrease in data de duplication factor.

EDIT: It also breaks the nightly catch all Backup, that always runs incase a new VM is not configured correctly or a backup is turned off temporarily.

 

[dcsapak]

We can configure purge policy in the cluster, But that will run purge and Garbage Collection as part of every backup ok, but then i can have multiple purge and GC processes running at once. This Causes us issues because we want to keep the last 4Hours of backups from the 30m group (keep-last=8), but don't want to add 8 Days to our 24h Group. I understand that the Purge&GC configs are kept in the vzdump file (I THINK)? Can PBS use those configs.

i don't completely understand what you mean here.. what is 'purge policy'?

you can defined the prune settings either in pve (per backup job) or on pbs (per datastore). the 'lowest' setting will always win
there should be no gc task started when setting the prune setting on pve side though, this is an individual setting of the datastore on pbs side?

so in you case i'd try with setting the prune config only on pve side and do a regular (but not too often) gc schedule on pbs

The next issue is that We have setup a PBS in AWS, It works. But we have no way of limiting what gets sent to the remote, We don't want backups being sent to the remote every 30 min just one daily, the backups are small so its not a huge deal. But seems like a waste of bandwidth.

again, not completely sure what you mean here. is that remote pbs pulling from your local one or do you backup directly there?
in either case, only the new chunks should be sent/pulled, so not much bandwidth should be wasted...