Remote sync slower than expected

[dignus]

We've been using PBS for over a year now, it meets all our needs, love it. However - somehow thee offsite sync is way slower than expected. We can't really find any obvious bottlenecks.

Benchmark for the source host:

Time per request: 6515 microseconds.
TLS speed: 643.70 MB/s
SHA256 speed: 1626.03 MB/s
Compression speed: 401.79 MB/s
Decompress speed: 496.67 MB/s
AES256/GCM speed: 1143.66 MB/s
Verify speed: 399.93 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 643.70 MB/s (52%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1626.03 MB/s (80%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 401.79 MB/s (53%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 496.67 MB/s (41%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 399.93 MB/s (53%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1143.66 MB/s (31%) │
└───────────────────────────────────┴────────────────────┘

Benchmark for the remote host:

Time per request: 4936 microseconds.
TLS speed: 849.60 MB/s
SHA256 speed: 575.39 MB/s
Compression speed: 537.77 MB/s
Decompress speed: 748.55 MB/s
AES256/GCM speed: 4345.56 MB/s
Verify speed: 312.47 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 849.60 MB/s (69%)   │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 575.39 MB/s (28%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 537.77 MB/s (72%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 748.55 MB/s (62%)   │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 312.47 MB/s (41%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 4345.56 MB/s (119%) │
└───────────────────────────────────┴─────────────────────┘

Source host hardware:
EPYC 7302P
64 GB RAM
ZFS pool striping across 2 raidZ pools build out of 5 Samsung PM983 8 TB NVME drives each

Remote host hardware:
Xeon E-2274G
64 GB RAM
ZFS pool striping across 2 raidZ pools build out of 12 Samsung PM863 1 or 2 TB drives each.

The connection in between is a 1 Gbit fiber.

As said - I cannot find any bottlenecks. CPU / RAM / disk / network / routers, nothing is heavily loaded. The sync speed is only 160 Mbit. This makes our sync run for about 16 hours, which soon won't be enough for sync'ing once a day.

Are there any tuneables?

 

[Chris]

Hi,

The connection in between is a 1 Gbit fiber.

As said - I cannot find any bottlenecks. CPU / RAM / disk / network / routers, nothing is heavily loaded. The sync speed is only 160 Mbit. This makes our sync run for about 16 hours, which soon won't be enough for sync'ing once a day.

what is the latency of the connection between sync source and target hosts? Is this a pull or a push sync? Please also share the sync task log. Most likely you are running into this issue [0].

Are there any tuneables?

At the moment there is no option, but there are patches on the mailing list, which implement parallel group syncs [1], that should help with throughput for high latency network connections.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182
[1] https://lore.proxmox.com/pbs-devel/20250120105104.206718-1-c.ebner@proxmox.com/T/

 

[dignus]

what is the latency of the connection between sync source and target hosts? Is this a pull or a push sync? Please also share the sync task log. Most likely you are running into this issue [0].

It's a pull. Latency is about 6 to 7 ms.

 

[Chris]

Can you please also share the sync job task log? How much data is being transferred?

 

[dignus]

Sure, see attachment. Just onder 1 TB.

 

[Chris]

You could try and see if the required sync transfer time is reduced if you create multiple sync jobs, scheduled at the same time and therefore running in parallel, but each with different group filters, e.g. by a regex based include filter to only selecting groups starting with a given prefix. By this you would effectively perform something similar to what the parallel sync feature would achieve.

I suggest to test this using a environment or at least us a dedicated testing namespace to sync in order to see if that truly helps to increase your throughput.