Remote Sync Pull very slow

dompro · 2026-10-21T07:19:42+0200

Hi,

we have a setup with 3 PBS Servers. A Primary for general purpose, a Secondary for big VMs (Fileserver) and a Remote that just pulls the backups from the Primary and Secondary.

Storage Primary:
XFS on Pure Storage FlashArray, FC multipath, 50 TB LUN

Code:

== Sequential WRITE ==
    slat (usec): min=17, max=416, avg=38.00, stdev=13.83
    clat (usec): min=2997, max=10408, avg=5204.37, stdev=857.17
     lat (usec): min=3042, max=10431, avg=5242.37, stdev=852.91
    clat percentiles (usec):
  lat (msec)   : 4=7.96%, 10=92.03%, 20=0.01%
  WRITE: bw=1499MiB/s (1572MB/s), 1499MiB/s-1499MiB/s (1572MB/s-1572MB/s), io=32.0GiB (34.4GB), run=21853-21853msec


== Sequential READ ==
    slat (usec): min=27, max=294, avg=48.00, stdev=11.51
    clat (usec): min=1642, max=7209, avg=3112.20, stdev=541.42
     lat (usec): min=1696, max=7262, avg=3160.20, stdev=539.95
    clat percentiles (usec):
  lat (msec)   : 2=0.02%, 4=93.05%, 10=6.93%
   READ: bw=2530MiB/s (2653MB/s), 2530MiB/s-2530MiB/s (2653MB/s-2653MB/s), io=32.0GiB (34.4GB), run=12953-12953msec

Storage Secondary:
ZFS, Mirror with 2 × raidz2 with 6 × 18.2 TB HDD each, 2 spares, no special vdev

Storage Remote:
ZFS dRAID draid2:6d:60c:4s, 60 × 8 TB HDD, no special vdev

Code:

== Sequential WRITE ==
    slat (usec): min=855, max=88075, avg=1577.24, stdev=3447.73
    clat (usec): min=3, max=201539, avg=12857.26, stdev=11014.67
     lat (usec): min=1028, max=204757, avg=14434.50, stdev=12044.32
    clat percentiles (msec):
  lat (usec)   : 4=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=16.32%, 20=77.80%, 50=4.47%
  lat (msec)   : 100=1.00%, 250=0.40%
  WRITE: bw=544MiB/s (571MB/s), 544MiB/s-544MiB/s (571MB/s-571MB/s), io=32.0GiB (34.4GB), run=60209-60209msec


== Sequential READ ==
    slat (usec): min=463, max=68074, avg=2283.89, stdev=3978.97
    clat (usec): min=5, max=151630, avg=16038.74, stdev=11463.71
     lat (usec): min=1395, max=163168, avg=18322.63, stdev=12363.18
    clat percentiles (msec):
  lat (usec)   : 10=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=47.16%, 20=27.36%, 50=23.52%
  lat (msec)   : 100=1.84%, 250=0.09%
   READ: bw=437MiB/s (458MB/s), 437MiB/s-437MiB/s (458MB/s-458MB/s), io=32.0GiB (34.4GB), run=75068-75068msec

RAM
Primary/Secondary 16G
Remote 256G

Network
The Primary and Secondary PBS have a 10G Uplink to the Remote. Tested with iperf.

Code:

[SUM]   0.00-30.00  sec  32.7 GBytes  9.36 Gbits/sec  5138             sender
[SUM]   0.00-30.00  sec  32.7 GBytes  9.36 Gbits/sec                  receiver

Local Benchmarks against the datastore

PBS Primary

Code:

Uploaded 1616 chunks in 5 seconds.
Time per request: 3099 microseconds.
TLS speed: 1353.32 MB/s
SHA256 speed: 1848.81 MB/s
Compression speed: 561.06 MB/s
Decompress speed: 870.20 MB/s
AES256/GCM speed: 10910.91 MB/s
Verify speed: 593.14 MB/s
┌───────────────────────────────────┬──────────────────────┐
│ Name                              │ Value                │
╞═══════════════════════════════════╪══════════════════════╡
│ TLS (maximal backup upload speed) │ 1353.32 MB/s (110%)  │
├───────────────────────────────────┼──────────────────────┤
│ SHA256 checksum computation speed │ 1848.81 MB/s (91%)   │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 compression speed    │ 561.06 MB/s (75%)    │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 decompression speed  │ 870.20 MB/s (73%)    │
├───────────────────────────────────┼──────────────────────┤
│ Chunk verification speed          │ 593.14 MB/s (78%)    │
├───────────────────────────────────┼──────────────────────┤
│ AES256 GCM encryption speed       │ 10910.91 MB/s (299%) │
└───────────────────────────────────┴──────────────────────┘

PBS Secondary

Code:

Uploaded 2078 chunks in 5 seconds.
Time per request: 2407 microseconds.
TLS speed: 1741.96 MB/s
SHA256 speed: 1851.47 MB/s
Compression speed: 576.47 MB/s
Decompress speed: 899.40 MB/s
AES256/GCM speed: 11323.67 MB/s
Verify speed: 605.19 MB/s
┌───────────────────────────────────┬──────────────────────┐
│ Name                              │ Value                │
╞═══════════════════════════════════╪══════════════════════╡
│ TLS (maximal backup upload speed) │ 1741.96 MB/s (141%)  │
├───────────────────────────────────┼──────────────────────┤
│ SHA256 checksum computation speed │ 1851.47 MB/s (92%)   │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 compression speed    │ 576.47 MB/s (77%)    │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 decompression speed  │ 899.40 MB/s (75%)    │
├───────────────────────────────────┼──────────────────────┤
│ Chunk verification speed          │ 605.19 MB/s (80%)    │
├───────────────────────────────────┼──────────────────────┤
│ AES256 GCM encryption speed       │ 11323.67 MB/s (311%) │
└───────────────────────────────────┴──────────────────────┘

PBS Remote

Code:

Uploaded 831 chunks in 5 seconds.
Time per request: 6028 microseconds.
TLS speed: 695.74 MB/s
SHA256 speed: 455.46 MB/s
Compression speed: 411.80 MB/s
Decompress speed: 653.62 MB/s
AES256/GCM speed: 2902.15 MB/s
Verify speed: 268.16 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 695.74 MB/s (56%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 455.46 MB/s (23%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 411.80 MB/s (55%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 653.62 MB/s (55%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 268.16 MB/s (35%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2902.15 MB/s (80%) │
└───────────────────────────────────┴────────────────────┘

Problem
The sync task runs very slow, giving anywhere from 1 to 10 chunks per second. Sync task is the only thing that runs, no other jobs running in that timeframe.

2026-10-19T19:18:26Z/drive-scsi0.img.fidx: downloaded 1.161 GiB (722.469 KiB/s)
2026-10-20T13:00:01Z/drive-scsi0.img.fidx: downloaded 1.247 GiB (4.457 MiB/s)
2026-10-20T19:22:03Z/drive-scsi0.img.fidx: downloaded 1.193 GiB (15.674 MiB/s)

From my understanding the sync should be faster as the backup. Since the backup has to compare which chunks he already has and which to write, but the sync already knows exactly which chunks he needs. Of course, that's only if the sync target is fast enough.

I know that the recommendation for PBS is to run SSD and way more RAM, but in the current economy we just couldn't splurge on that. We didn't expect line-speed for the sync, but we also didn't expect to see KiB/s in the log.

PS: For transparency, I also opened a ticket for this, but I wanted to hear the communities opinion on this too.

robertlukan · 2026-10-21T07:52:41+0200

Can you elaborate how far away is the remote PBS from the main PBS, in terms of latency ?

dompro · 2026-10-21T08:20:27+0200

Connected over about 400m Singlemode with a single switch in between.
ICMP: time=0.191 ms

MarkusKo · 2026-10-21T09:10:11+0200

Code:

I know that the recommendation for PBS is to run SSD and way more RAM, but in the current economy we just couldn't splurge on that.

PBS itself does not need much RAM. If you use ZFS then you can use spare RAM for the ARC cache but if you don't have much RAM, reduce ARC cache.

I run a virtualized PBS on a NAS (ext4 filesystem) that pulls from the primary bare metal PBS and it uses ~ 2,2GB RAM. Sure, i have a small setup but 256GB for PBS seems overkill ...

fabian · 2026-10-21T09:22:01+0200

could you provide the full task logs? what about host metrics on both ends?

robertlukan · 2026-10-21T09:55:59+0200

dompro said:
Connected over about 400m Singlemode with a single switch in between.
ICMP: time=0.191 ms

Install iperf3 and test the network performance, just to confirm that you have the bandwidth.

dompro · 2026-10-21T10:55:04+0200

fabian said:
could you provide the full task logs? what about host metrics on both ends?

Attached is one of the Task Logs. I can provide more, but to my eye they all look quite similar. This sync was quite large because we added a 20TB fileserver to the backup previously to this sync.

Regarding metrics. I let claude write me a script that gives back the median of 4 hour timebuckets from the PBS API Results. Since the Job ran 15 days giving every single datapoint would be quite a lot. But I can provide that if needed.

dompro · 2026-10-22T06:18:19+0200

robertlukan said:
Install iperf3 and test the network performance, just to confirm that you have the bandwidth.

Hi, the iperf result is in my initial post. Is that enough or do you want to see the full result?

fabian · 2026-10-22T08:39:56+0200

which version of PBS are you running (on each of the three systems)?

with "metrics" I meant
- start a sync
- look on both ends how the CPU/memory/IO situation looks like
- check if any particular threads or processes or disks are pegged

Remote Sync Pull very slow

dompro

Member

robertlukan

Member

dompro

Member

MarkusKo

Renowned Member

fabian

Proxmox Staff Member

robertlukan

Member

dompro

Member

Attachments

dompro

Member

fabian

Proxmox Staff Member

We value your privacy