Remote sync extremely slow

C

Colin 't Hart

Renowned Member
Jan 20, 2017
81
14
73
53
Frösön, Sweden
hiawathaavenue.com
Hi,

We're wanting to replace a somewhat home-grown backup solution based on rsnapshot and rsync with Proxmox Backup Server. The rsnapshot solution uses rsync to mirror the backups to other servers, including one off-site.

I have Proxmox Backup Server configured as a remote but it's syncing extremely slowly. It has transferred 30MB in the last day or so; my backups are only about 400GB at present, expecting to go up to close to 4TB. This is more than we have at present since we can now backup entire VMs whereas the current solution only backs up the actual data.

CPU usage during syncing is extremely low, hovering at around 25% of one core. As far as I can tell I have not enabled any throttling, encryption, compression.

I there any difference in speed between push and pull?

Why is this syncing so much slower than rsync on identical hardware?
 
There is a speed limit in the job settings but it defaults to none/off.

Our speeds jumped notably when enabling FQ Codel on the connection even though it was syncing overnight. Do you by chance have a limiter in your router at either end?
 
SteveITS said:
There is a speed limit in the job settings but it defaults to none/off.

Our speeds jumped notably when enabling FQ Codel on the connection even though it was syncing overnight. Do you by chance have a limiter in your router at either end?
Click to expand...
No limiting in the routers at each end. They are both EdgeRouter X. The overall speed is limited by one end being 100mbit. We've never noticed any sort of slow speeds in these fantastic little routers.
I notice that it supports Smart Queue Management and FQ_CODEL. Any tips on configuring this?
 
No idea on that model, sorry. We're using pfSense so on the server/data center router we set a limiters with different Night (currently 90 Mbps) and Day (20) bandwidth limits, with FQ_CODEL as the Scheduler. The reported speed per PBS logs went from something like 5-6 MB/s (not Mbps) to around 10, as I recall.
 
I just realised that all the traffic is double-encrypted as it's going over inter-site VPN too. I can make the traffic go directly over the Internet and see if that helps.

Currently it's receiving about 24mbit.

Right now I'm pulling on the remote; is pushing on the primary better than pulling?
 
Last edited:
I'm still not convinced the current sync protocol can't be improved to take advantage of rsync-like smarts. But maybe it's just me not being used to how long the initial syncs take.

I did manage to find a few areas for improvement in our infra though.

Turns out the remote site was using an old 100-mbit switch because they needed PoE support for some devices and everything was just connected to it.
I moved the remote servers directly to some spare (gigabit) ports on the EdgeRouter X and discovered our connection there is capped at around 300mbit, versus the 100mbit I thought it was.

Secondly I added some NAT and firewall rules and can now safely route the PBS HTTPS traffic directly over the Internet.

Now I'm seeing 250mbit sync rates versus the 25mbit I was seeing before.

I'm a happy camper.
 
Nice and frustrating find. I hate it when those old things creep in.

For the initial sync, in the job's advanced settings there is a field for the number of backups of each VM to sync, as in, the last 2 backups instead of 300 for VM1, then 300 for VM2...
 
SteveITS said:
AFAIK the pull/push direction has no bearing on speed. It’s more a function of design/security. A pull job doesn’t remove anything for example.
Click to expand...
I disagree on this one.
The push job can not specify how many workers should be used, so it is using only one. With the pull job you can specify how many workers are used and copy multiple backups at the same time, saturating the network connection more.
Also, the documentation here https://pbs.proxmox.com/docs/managing-remotes.html#sync-direction-push makes a distinction between push and pull, with pull being more versatile.
 
  • Like
Reactions: SteveITS
SteveITS said:
Nice and frustrating find. I hate it when those old things creep in.

For the initial sync, in the job's advanced settings there is a field for the number of backups of each VM to sync, as in, the last 2 backups instead of 300 for VM1, then 300 for VM2...
Click to expand...
There is a caveat to this method. If you want to sync the additional backups later, you have to jump through hoops to get them going.
I just did a sync from an old PBS to a new one and wanted to have the latest backup first (in case the old PBS dies), but when I attempted to run the sync job again, it would skip the older backups. Turns out that PBS checks which backup is the newest on the server to sync to and skips all older backups.
To circumvent that, you have to create a namespace and sync the backups into that on the new (or remote) PBS.
The only bright spot of this method is, that it still deduplicates with your previously synced (newer) backups, so no extra space is wasted.
 
You must log in or register to reply here.
Top Bottom
Back