PBR Remote Sync - Limiting the sync

[grepler]

Our main backup server has 20+TBs of available storage (our colo server), while our offsite only has 12TB (public bare metal cloud).
Since I can't easily get additional storage at my offsite location and I only need to offsite store a smaller set of last-resort backups (say, the 5 most recent and 2 annual), is there a way to only sync the 5 most recent backups per VM, for example?

I can set a more aggressive prune policy at my offsite location to keep the storage usage in line, but as I understand it, every time it tries to pull the primary datastore it will pull it's entire contents, event the backups which would be immediately pruned at the destination.

 

[grepler]

One method I was considering (but it means I lose some de-duplication) is to have a 'Target' datastore and a 'Deep Archive' datastore at the primary site.
PVEs backup to the target, which keeps the last 2-5, then does a local sync to Deep Archive. then offsite runs a daily sync job to pull from Target.

Anyone have experience with this kind of setup?

 

[dietmar]

I can set a more aggressive prune policy at my offsite location to keep the storage usage in line, but as I understand it, every time it tries to pull the primary datastore it will pull it's entire contents, event the backups which would be immediately pruned at the destination.

No, it only sync backups newer than the last local backup.

 

[grepler]

Oh! that's excellent news - but how would I get the initial sync (I.E. copy the latest set of backups) to the datastore, if a full sync will fill the destination drives? @dietmar

 

[dietmar]

Oh! that's excellent news - but how would I get the initial sync (I.E. copy the latest set of backups) to the datastore, if a full sync will fill the destination drives? @dietmar

We are working on such feature ...

 

[grepler]

perhaps I could fool it by copying some of these directories? even if the verifications fail and I delete them afterwards, would that serve as a starting point for a given vm?

 

[dietmar]

perhaps I could fool it by copying some of these directories?

Sorry, I never tested that. But you can simpy try yourself - I guess that may work.

 

[grepler]

For any future users, I did in fact SSH from my off-site server to my primary backup server, copied the /vm and /ct folders in the datastore to my new server, setup a more aggressive prune rule and then started replication.

It starts transferring only the newest images, and once you have a few transferred over, kick off a verification task on the datastore. It will mark as failed any of the 'fake' folders you copied over via ssh, and then you can delete the old items.