PBS 4.2 and S3: Ways to keep a second copy without bucket versioning?

[sergioposadam]

Hi all,

I'm designing offsite backups from PBS to AWS S3 for several sites, some with limited internet uplinks. My goal is that PBS uploads each backup once to a primary S3 bucket, and the second copy is created within AWS, I specifically want to avoid the second copy going over the site link.

Environment

• Proxmox Backup Server 4.2, S3 storage backend
• AWS S3 primary bucket, SSE-KMS enforced, HTTPS via outbound proxy, datastore under a bucket prefix.
• PBS client-side encryption enabled.
• Several remote sites with constrained bandwidth

My understanding so far (please correct me)

• PBS does not support S3 versioning, because it manages chunk lifecycle itself and versioning breaks Garbage collectoin
• AWS S3 replication (CRR/SRR) requires versioning on both buckets, so I can't use native S3 replication to mirror the PBS datastore bucket to a second bucket.
• Based on 3-2-1-style setup:

My specific questions

1. Is the "no versioning" guidance mainly about Object Lock, or does versioning by itself also break PBS? If PBS deletes simply create delete markers, GC still succeeds functionally. Is the only real downside accumulating noncurrent versions?
2. If so, has anyone run a PBS S3 datastore on a versioning-enabled bucket specifically to enable CRR/SRR,and is that considered safe?
3. If versioning is genuinely unsafe for PBS, what's the recommended way to get a cloud side second copy without pulling chunks back through an on site PBS? For example:
   * S3 Batch Operations Copy (Inventory-driven, works on non-versioned buckets) on a schedule?
   * A second PBS instance running in AWS doing a sync job between two S3-backed datastores in different regions?
4. For anyone doing this in production on 4.2 at bandwidth constrained sites — what topology did you use, and what should I avoid?
   

I want the replication to happen on AWS, not the site uplink. Trying to confirm whether CRR/SRR is achievable with PBS or whether I should design around Batch Copy or cloud PBS instead.

 

[Johannes S]

Do I understand correctly that the several remote sites each have a ProxmoxVE installation but you have only one PBS which backups everything to S3? Then I would setup a local PBS in each of them, especially if their bandwith is limited and only use S3 as additional offsite copy (and to have one Backup on different media).
This has following benefits:

• Restores will be faster than pulling everything via S3 and remote PBS
• PBS can sync between different PBS, so you could use the PBS with the best uplink to also pull backups from the rest (this could even be seeded with external USB storage as removable datastore so the initial sync/backups won't waste your bandwith but will be done with the aid of the removable datastore). After the inital backups/syncs this shouldn't cost to much bandwith since only changes will be uploaded

If I understand wrong and you already have local PBS then I fear that what you want isn't actually possible (but will be happy to be prooven wrong):

If versioning is genuinely unsafe for PBS, what's the recommended way to get a cloud side second copy without pulling chunks back through an on site PBS?

A sync job will always just transfer chunks not already present on the target. So you need (no way around it) a way to copy the chunks in a safe way. In the past (before PBS S3-support was introduced) people did rather nasty hacks by manually syncing their datastores to S3 with rclone or s3fs. Depending on the way the people implemented these they could broke backups (see https://forum.proxmox.com/threads/datastore-synced-with-rclone-broken.154709/ ) since according to PBS developers the sync mechanism must take care, that the chunks are transferred correctly in the correct order. Now I'm not sure whether this still apply to native S3-backed datastores, would be nice to have confirmation from staff on this. To get this a feature request for supporting S3 CRR/SRR on bugzilla.proxmox.com might be helpful (since there is no guarantee whether developers will read everything on this community forum but they monitor their bugzilla for new bugs and feature/requests) to getting to know whether that usecase is supported or not.

A second PBS instance running in AWS doing a sync job between two S3-backed datastores in different regions?

This is propably the best course of action at the moment. I'm not sure about the envolved transfer costs though. I thought AWS charges for transfers between different regions?

 

[sergioposadam]

Hi Johannes,

We already run a local PBS per site and take local backups, but some sites don't have enough local storage for VMs and backups, so S3 is our offsite copy, problem is the site links are thin.

The thing I'd love a dev answer on (I'll file the bugzilla request): the versioning question. Object Lock is clearly fatal, it blocks deletes so GC can't reclaim.

Versioning I haven't tested yet, but on paper it seems harmless: PBS deletes should still succeed (S3 just drops a delete marker), GC keeps working, and the only cost is noncurrent versions piling up

Also, I read you don't really need a 2nd PBS for the having 2 copies. A single PBS can hold 2 S3 datastores (bucket A + B) and run a local sync job between them. If that PBS lives in the cloud, A to B copy stays AWS, so, no site bandwidth used.

So, is versioning actually unsafe for a datastore, or just a cost thing? If it's just cost, CRR/SRR becomes a clean way to get the offsite copy with no second PBS at all.

Thanks!

 

[Johannes S]

Can you please link the bugzilla ticket after creating it? I would be interested in an answer as well.