Remote sync slower than expected

[dignus]

We've been using PBS for over a year now, it meets all our needs, love it. However - somehow thee offsite sync is way slower than expected. We can't really find any obvious bottlenecks.

Benchmark for the source host:

Time per request: 6515 microseconds.
TLS speed: 643.70 MB/s
SHA256 speed: 1626.03 MB/s
Compression speed: 401.79 MB/s
Decompress speed: 496.67 MB/s
AES256/GCM speed: 1143.66 MB/s
Verify speed: 399.93 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 643.70 MB/s (52%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1626.03 MB/s (80%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 401.79 MB/s (53%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 496.67 MB/s (41%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 399.93 MB/s (53%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1143.66 MB/s (31%) │
└───────────────────────────────────┴────────────────────┘

Benchmark for the remote host:

Time per request: 4936 microseconds.
TLS speed: 849.60 MB/s
SHA256 speed: 575.39 MB/s
Compression speed: 537.77 MB/s
Decompress speed: 748.55 MB/s
AES256/GCM speed: 4345.56 MB/s
Verify speed: 312.47 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 849.60 MB/s (69%)   │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 575.39 MB/s (28%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 537.77 MB/s (72%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 748.55 MB/s (62%)   │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 312.47 MB/s (41%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 4345.56 MB/s (119%) │
└───────────────────────────────────┴─────────────────────┘

Source host hardware:
EPYC 7302P
64 GB RAM
ZFS pool striping across 2 raidZ pools build out of 5 Samsung PM983 8 TB NVME drives each

Remote host hardware:
Xeon E-2274G
64 GB RAM
ZFS pool striping across 2 raidZ pools build out of 12 Samsung PM863 1 or 2 TB drives each.

The connection in between is a 1 Gbit fiber.

As said - I cannot find any bottlenecks. CPU / RAM / disk / network / routers, nothing is heavily loaded. The sync speed is only 160 Mbit. This makes our sync run for about 16 hours, which soon won't be enough for sync'ing once a day.

Are there any tuneables?

 

[Chris]

Hi,

The connection in between is a 1 Gbit fiber.

As said - I cannot find any bottlenecks. CPU / RAM / disk / network / routers, nothing is heavily loaded. The sync speed is only 160 Mbit. This makes our sync run for about 16 hours, which soon won't be enough for sync'ing once a day.

what is the latency of the connection between sync source and target hosts? Is this a pull or a push sync? Please also share the sync task log. Most likely you are running into this issue [0].

Are there any tuneables?

At the moment there is no option, but there are patches on the mailing list, which implement parallel group syncs [1], that should help with throughput for high latency network connections.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182
[1] https://lore.proxmox.com/pbs-devel/20250120105104.206718-1-c.ebner@proxmox.com/T/

 

[dignus]

what is the latency of the connection between sync source and target hosts? Is this a pull or a push sync? Please also share the sync task log. Most likely you are running into this issue [0].

It's a pull. Latency is about 6 to 7 ms.

 

[Chris]

Can you please also share the sync job task log? How much data is being transferred?

 

[dignus]

Sure, see attachment. Just onder 1 TB.

 

[Chris]

You could try and see if the required sync transfer time is reduced if you create multiple sync jobs, scheduled at the same time and therefore running in parallel, but each with different group filters, e.g. by a regex based include filter to only selecting groups starting with a given prefix. By this you would effectively perform something similar to what the parallel sync feature would achieve.

I suggest to test this using a environment or at least us a dedicated testing namespace to sync in order to see if that truly helps to increase your throughput.

 

[dignus]

Little ashamed to say the issue was found and was not in PBS. The ipsec tunnel endpoints had some issues. Now that these are resolved we can completely fill the gbit connection.

 

[Chris]

Glad to hear that you found the issue and were able to resolve this! Can you share the exact cause and how you fixed it for others to find a solution more easily? That would be great, thanks.

 

[dignus]

Sure. We found out one core in the ipsec endpoint (pfsense) on one side was running at 100% load and was limiting the transfer speed. After enabling MSS clamping (preventing fragmentation) and Asynchronous Cryptography (use multiple cores for multiple ipsec functions) transfer speed started to max out the connection.