Backup strategy for sync and retention with two pbs

[Tobias.F]

Hello,

I am using pbs since a while and now start to extend my local pbs with an additional remote pbs.
Now I am struggling to define the proper place to do retention when syncing from one pbs to the other.

Current use case:
cloud vps with proxmox backup client --> cloud pbs --> local pbs

backup
The backup client is doing backups on hourly base. It is jus a small inkremental.
I am backing to the cloud pbs, because I would like to avoid to push the data twice through my home internet connection when doing sync.

retention
I would like to end up with something like this:
- hourly: 24 per day
- daily: 7 per week
- weekly: 5 per month
- monthly: 6 year
- yearly: 5

Strategy
I do not want to do the retention by the client (vps with backup client)
I want to do syn by pull, not push strategy.


Question:
How to combine retention and sync in the best way and where to do the retention?
Is retention by cloud pbs only and sync to local pbs with option varnish a good idea?

 

[tcabernoch]

Here's how I set things up.

My hosting is local. The VMs run on hardware that lives here.
(Where are your VMs running, local or cloud?)

Hardware PBS does the backups. This corresponds to your local PBS.
This machine is closest to the data, runs backups the fastest, and has the least impact on the VMs being backed up.

Virtual PBS does the large capacity offsite data storage. This corresponds to your cloud PBS.
This virtual machine is closest to the large storage, where it can write and access that data quickly.
The virtual PBS runs a Pull Sync Job to get the backups from the local PBS.
I also use the virtual PBS as a hub for site-to-site Sync with other offsite locations.

So for me it looks like ... Local PVE > Local hardware PBS > Cloud virtual PBS > and then > Offsite cloud virtual PBS

 

[Tobias.F]

My hosting is local. The VMs run on hardware that lives here.
(Where are your VMs running, local or cloud?)

Both. VMs local and vps in the cloud using the same backup infrastruckture.
Not to confuse in the discussion, I limited to my question to only one use case: vps in the cloud with Proxmox Backup Client installed on it.

So for me it looks like ... Local PVE > Local hardware PBS > Cloud virtual PBS > and then > Offsite cloud virtual PBS

Now I would be interested in how you have realized retention in this chain.

 

[tcabernoch]

Now I would be interested in how you have realized retention in this chain.

The local hardware PBS (where SSD is a scarce resource) has a short retention of 2 days or so.

The virtual cloud PBS (with the large storage) retain for a couple weeks.

The offsite virtual cloud PBS (in the remote geographic location) has a retention of 1 day, just the latest backup for DR.

We provide a longterm storage service as well, which is simply a backup export and upload to our S3 bucket, typically for machines being decomm that they just want to keep around.

 

[Tobias.F]

The local hardware PBS (where SSD is a scarce resource) has a short retention of 2 days or so.
The virtual cloud PBS (with the large storage) retain for a couple weeks.
The offsite virtual cloud PBS (in the remote geographic location) has a retention of 1 day, just the latest backup for DR.

Do I get it right?
Each of the 3 pbs is running the retention job for itself, just with its individual setting for its own retention? You do not make use of any varnish function or recusion logic?

 

[tcabernoch]

Do I get it right?
Each of the 3 pbs is running the retention job for itself, just with its individual setting for its own retention? You do not make use of any varnish function or recusion logic?

I utilize a series of Namespaces containing colo name, functional group, and back or sync.

chicago-dev/back - These backups were made at this site. Local retention strategies are applied.
chicago-dev/sync - These backups were synced from a remote site. 1 day retention strategy is applied.

I use recursion as needed. I don't use vanish at all.

 

[SteveITS]

The remote PBS pulls the backups. Both PBS do their own retention. Thus the remote can have a longer retention.

The node doing the backups doesn’t have the rights to delete backups.