Limit number of backups sent to remote (off-site) backup server

[Tmanok]

Hi Everyone,

Perhaps I've missed something in the forum or in the manual, but is there a way to reduce or limit the number of backups sent to a remote PBS?
For example, say my on-site backup server had 8TiB and was used for A LOT of backups, but my off-site was say 4TiB usable, could the sync job specify what to pull from the larger PBS instance to the smaller off-site instance?

My workplace does not have this issue, but I am anticipating it in a future personal setup. When I look at what I setup at my work, I don't see any options for this (see attached).
Thank you,


Tmanok

 

[fabian]

there will be some more filtering options to limit which groups are synced (based on type, id, or a custom regex). other than that, you can set up more aggressive pruning on the sync target - that won't stop you from transferring lots of data, but reduces the amount stored long-term

 

[Tmanok]

Hey Fabian,

Ok I actually quite like that idea! It changes how I perceive the structure a bit too. My original understanding that an off-site server pulling from an on-site server would simply mirror the on-site server and not have to do its own garbage collection. I've since setup garbage collection on the off-site server (which is new I might add), and I've tuned it to be more aggressive than the on-site server.

Although I am a bit concerned about wasting WAN throughput, I'm not concerned about transferring less than a TB a week.

there will be some more filtering options

Meaning it is in the roadmap for future releases? That would be fantastic! I've also considered the thought that there could be a secondary ons-site proxmox backup server that is used as a repository for a different PVE backup schedule, at that point it would be easy to make fewer backups. Then you could choose to only backup the secondary on-site backup server to the off-site backup server. Maybe a bit of wasted disk and an extra VM, but at least your off-site could be much smaller (if that was your goal).

Thanks Fabian!


Tmanok

 

[fabian]

Meaning it is in the roadmap for future releases? That would be fantastic! I've also considered the thought that there could be a secondary ons-site proxmox backup server that is used as a repository for a different PVE backup schedule, at that point it would be easy to make fewer backups. Then you could choose to only backup the secondary on-site backup server to the off-site backup server. Maybe a bit of wasted disk and an extra VM, but at least your off-site could be much smaller (if that was your goal).

the patches for group filtering have been through two iterations already, I'll send a (hopefully final ) version soon.

https://lists.proxmox.com/pipermail/pve-devel/2021-September/049998.html

 

[Tmanok]

two iterations already

If they aren't ready, then they just aren't ready. I believe in the quality that the Proxmox team puts into their products, and I'm happy to wait for a quality product, especially one that you're proud of.