Remote sync taking forever

[Comrade7928]

Hello, I have a Proxmox Backup Server which all my VMs are backed up to and another offsite backup server which gets synced by the local one.
My problem is, one of the VMs has a 16TB disk attached and the remote sync takes forever to sync it. I understand the bandwidth over the internet isnt the fastest, however in the backup logs it shows only 10-30GB added on this volume per backup on average but the sync still takes the same amount of time for all backups.

I figure the sync has to check all chunks even if they already exist in the target, but because of this the remote sync takes around 6-7 days usually, during which the new backups that are done daily add up, so the remote Backup server ends up very far behind in the backups it has.

One idea I had was to just rsync the datastores with --ignore-existing so it really only has to upload the new chunks and backups but I'm not sure how well that would work.
Any ideas how to optimize this?

 

[Chris]

Hello, I have a Proxmox Backup Server which all my VMs are backed up to and another offsite backup server which gets synced by the local one.
My problem is, one of the VMs has a 16TB disk attached and the remote sync takes forever to sync it. I understand the bandwidth over the internet isnt the fastest, however in the backup logs it shows only 10-30GB added on this volume per backup on average but the sync still takes the same amount of time for all backups.

I figure the sync has to check all chunks even if they already exist in the target, but because of this the remote sync takes around 6-7 days usually, during which the new backups that are done daily add up, so the remote Backup server ends up very far behind in the backups it has.

One idea I had was to just rsync the datastores with --ignore-existing so it really only has to upload the new chunks and backups but I'm not sure how well that would work.
Any ideas how to optimize this?

Hi,
the sync job in push direction follows the same incremental logic as a regular backup, so only new chunks will be synced. Can you share the corresponding backup task logs on the target PBS host? That should tell if the previous backup snapshot was used to track known chunks. What transfer speeds do you expect? Did you test you network speed?

 

[Comrade7928]

Hi,
the network speed is a maximum of 45 Mbit/s, the speed test says around 40 Mbits/s and I have limited the sync to 2 MegaBYTES/s to not clog up the entire upload we have.
From what i gather from the logs the bandwidth is around 2 MB/s so that adds up.
Attached are the logs from the sync. It does look like the target pbs only downloads the new chunks since the "downloaded" is 1.241 TiB but it still takes forever (the sync from the attached log took around 8 days). The backup I am referring to is vm 201 drive scsi 2.

I will try to set up a push sync after the currently running task is finished, thank you for the suggestion.

 

[Chris]

But than there is nothing wrong here? You can only download as fast as your rate limit or internet speed will allow. Also a sync job in the other direction will not change that and an rsync is also limited by your connection speed.

The thing is, if your deltas are small, the next sync should be faster, as it only has to sync unknown chunks. What was the actual delta between vm/201/2025-02-23T10:00:43Z and vm/201/2025-02-24T10:44:52Z? You should see that from the backup task log of the backup run creating snapshot vm/201/2025-02-24T10:44:52Z