Remote Syncing to Offsite

[Carl_Fitz_93]

I have 2 PBS Servers at two different physical locations. Each PBS server will complete it's own local backup from the Proxmox Cluster it has running on the same LAN. This is extremely quick as we have it set up on a 10Gb Switch on it's own network.

Now, I want to take that local backup and remote sync it to the PBS that is physically at our different site, I've set up the remote PBS server and I'm completing the sync. Where deduplication works for the initial backup whereby it will only backup the changes, so a 500Gb VM will only sync 2Gb of changes on the backup. When pushing the backups to it's remote sync, it appears to be syncing the full drive SCSI each time of 500Gb rather than just the changes, then it completes deduplication when it's on the other side? This makes syncing the remote backups take far too long and they run into the back of each other i.e. they don't complete in the 1 day window.

Is there any way around making remote syncs quicker?

 

[Dunuin]

Doing the same here (two PBS syncing each other) and it only transfers the chunks that are missing.
Is there anything interesting in the log of the sync task?

 

[Carl_Fitz_93]

The sync task in the log is suggesting it is only syncing the changes as the the data is so small that it is downloading

2023-05-31T23:11:34+01:00: percentage done: 1.94% (0/20 groups, 7/18 snapshots in group #1)
2023-05-31T23:11:34+01:00: sync snapshot ct/100/2023-05-22T16:40:03Z
2023-05-31T23:11:34+01:00: sync archive pct.conf.blob
2023-05-31T23:11:34+01:00: sync archive root.pxar.didx
2023-05-31T23:18:19+01:00: downloaded 3862376306 bytes (9.10 MiB/s)
2023-05-31T23:18:19+01:00: sync archive catalog.pcat1.didx
2023-05-31T23:18:19+01:00: downloaded 138196 bytes (1.87 MiB/s)
2023-05-31T23:18:19+01:00: got backup log file "client.log.blob"
2023-05-31T23:18:19+01:00: sync snapshot ct/100/2023-05-22T16:40:03Z done
2023-05-31T23:18:19+01:00: percentage done: 2.22% (0/20 groups, 8/18 snapshots in group #1)
2023-05-31T23:18:19+01:00: sync snapshot ct/100/2023-05-23T16:40:00Z
2023-05-31T23:18:19+01:00: sync archive pct.conf.blob
2023-05-31T23:18:19+01:00: sync archive root.pxar.didx
2023-05-31T23:25:24+01:00: downloaded 4241697382 bytes (9.53 MiB/s)
2023-05-31T23:25:24+01:00: sync archive catalog.pcat1.didx
2023-05-31T23:25:24+01:00: downloaded 138201 bytes (2.46 MiB/s)
2023-05-31T23:25:24+01:00: got backup log file "client.log.blob"
2023-05-31T23:25:24+01:00: sync snapshot ct/100/2023-05-23T16:40:00Z done
2023-05-31T23:25:24+01:00: percentage done: 2.50% (0/20 groups, 9/18 snapshots in group #1)
2023-05-31T23:25:24+01:00: sync snapshot ct/100/2023-05-24T16:40:02Z
2023-05-31T23:25:24+01:00: sync archive pct.conf.blob
2023-05-31T23:25:24+01:00: sync archive root.pxar.didx
2023-05-31T23:33:16+01:00: downloaded 4789113443 bytes (9.68 MiB/s)
2023-05-31T23:33:16+01:00: sync archive catalog.pcat1.didx
2023-05-31T23:33:16+01:00: downloaded 554345 bytes (3.24 MiB/s)
2023-05-31T23:33:16+01:00: got backup log file "client.log.blob"
2023-05-31T23:33:16+01:00: sync snapshot ct/100/2023-05-24T16:40:02Z done
2023-05-31T23:33:16+01:00: percentage done: 2.78% (0/20 groups, 10/18 snapshots in group #1)

However, the traffic log on my Firewall indicates that so much more traffic has been sent to that machine during the sync process

 

[drjaymz@]

Maybe the initial sync was just a major pain. My understanding is that it should work out what is missing from the remote and only copy the differences. If thats not the case then its really not fit for purpose and you'd be better using rsync using linking.

However, it looks like this may be working as intended. Having spoke to Carl.

Any tips greatly appreciated, especially anyone else using it like this.

What we want is a local backup and a remote backup and the remote backup is bandwidth limited < 100M and it needs to play nicely with everything else on that link. I know that we can set traffic policies to limit it to 50% of the available if required. 100M is a lot slower that you think it is, we forget how slow that is!

 

[fabian]

https://git.proxmox.com/?p=proxmox-...9bfd07f4c41c092a290bd7f9c8b7cb14;hb=HEAD#l174

it will definitely only download the chunks that are not already existing locally (note the early return in line 180).

the only things that are "redownloaded" for every sync are
- the remote snapshot list (there is no way to do a partial query for this information, and we need the full list for "remove_vanished" support anyway)
- the manifest and possibly index and log files of the last local snapshot in each group, if it exists remotely (this is indicated as "re-sync" in the log)

note that the manifest and index files are rather small, although index files obviously scale with the size of the corresponding backup archive/image, so if you backup big amounts of data, then individual index files might approach or even overtake a single chunk in size

 

[drjaymz@]

https://git.proxmox.com/?p=proxmox-...9bfd07f4c41c092a290bd7f9c8b7cb14;hb=HEAD#l174

it will definitely only download the chunks that are not already existing locally (note the early return in line 180).

the only things that are "redownloaded" for every sync are
- the remote snapshot list (there is no way to do a partial query for this information, and we need the full list for "remove_vanished" support anyway)
- the manifest and possibly index and log files of the last local snapshot in each group, if it exists remotely (this is indicated as "re-sync" in the log)

note that the manifest and index files are rather small, although index files obviously scale with the size of the corresponding backup archive/image, so if you backup big amounts of data, then individual index files might approach or even overtake a single chunk in size

This has to be miles more efficient that the legacy homebrew solution we were using which was rsync based.
One of the challenges with such a system is that a long sync (> 1 hour) invariably encounters some brief interruption or bandwidth issues, at which point the command will stall and another won't start whilst its stalled. That and it was copying 4Gb a day. The actual delta is much less. Hopefully this solution will do the job and be far easier to work out what went wrong if it did.

 

[fabian]

pulling also doesn't like the network connection being interrupted, but as long as no GC is run in parallel (and even then, the interruption between pulls must be long, or the changes happening in between must be big for it to have negative effects), chunks of partially downloaded snapshots are kept and the next sync will basically pick up where it left off.

 

[markf1301]

What did everyone determine was the best resolution? Having copies of your 'local' backups offsite is good, but what if you need to 'restore' those servers at the remote site? The backup from the 'local' environment isn't restorable from the remote sides' backup server..is it?

 

[drjaymz@]

What did everyone determine was the best resolution? Having copies of your 'local' backups offsite is good, but what if you need to 'restore' those servers at the remote site? The backup from the 'local' environment isn't restorable from the remote sides' backup server..is it?

You can mount any backup server anywhere and restore it. I literally do that all the time. Unless I misunderstood your question?

If you mean you can't see the remote namespace locally, then no - but you can go to datacentre and add your local backup and this time specify the namespace you need and it will then show all of them and you can restore.

 

[Dunuin]

The backup from the 'local' environment isn't restorable from the remote sides' backup server..is it?

No. But you could sync all backup snapshots at very short intervals so everything is available at both locations.
Or like already said, you could add both PBS to all PVEs to do remote restores. Add one PBS with a restore-only privileged user and this isn't even a problem for ransomware/human error protection.