Remote Sync Pull very slow

D

dompro

Member
Proxmox Subscriber
Oct 30, 2025
41
3
8
Hi,

we have a setup with 3 PBS Servers. A Primary for general purpose, a Secondary for big VMs (Fileserver) and a Remote that just pulls the backups from the Primary and Secondary.

Storage Primary:
XFS on Pure Storage FlashArray, FC multipath, 50 TB LUN

Code: 
== Sequential WRITE ==
    slat (usec): min=17, max=416, avg=38.00, stdev=13.83
    clat (usec): min=2997, max=10408, avg=5204.37, stdev=857.17
     lat (usec): min=3042, max=10431, avg=5242.37, stdev=852.91
    clat percentiles (usec):
  lat (msec)   : 4=7.96%, 10=92.03%, 20=0.01%
  WRITE: bw=1499MiB/s (1572MB/s), 1499MiB/s-1499MiB/s (1572MB/s-1572MB/s), io=32.0GiB (34.4GB), run=21853-21853msec


== Sequential READ ==
    slat (usec): min=27, max=294, avg=48.00, stdev=11.51
    clat (usec): min=1642, max=7209, avg=3112.20, stdev=541.42
     lat (usec): min=1696, max=7262, avg=3160.20, stdev=539.95
    clat percentiles (usec):
  lat (msec)   : 2=0.02%, 4=93.05%, 10=6.93%
   READ: bw=2530MiB/s (2653MB/s), 2530MiB/s-2530MiB/s (2653MB/s-2653MB/s), io=32.0GiB (34.4GB), run=12953-12953msec


Storage Secondary:
ZFS, Mirror with 2 × raidz2 with 6 × 18.2 TB HDD each, 2 spares, no special vdev

Storage Remote:
ZFS dRAID draid2:6d:60c:4s, 60 × 8 TB HDD, no special vdev

Code: 
== Sequential WRITE ==
    slat (usec): min=855, max=88075, avg=1577.24, stdev=3447.73
    clat (usec): min=3, max=201539, avg=12857.26, stdev=11014.67
     lat (usec): min=1028, max=204757, avg=14434.50, stdev=12044.32
    clat percentiles (msec):
  lat (usec)   : 4=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=16.32%, 20=77.80%, 50=4.47%
  lat (msec)   : 100=1.00%, 250=0.40%
  WRITE: bw=544MiB/s (571MB/s), 544MiB/s-544MiB/s (571MB/s-571MB/s), io=32.0GiB (34.4GB), run=60209-60209msec


== Sequential READ ==
    slat (usec): min=463, max=68074, avg=2283.89, stdev=3978.97
    clat (usec): min=5, max=151630, avg=16038.74, stdev=11463.71
     lat (usec): min=1395, max=163168, avg=18322.63, stdev=12363.18
    clat percentiles (msec):
  lat (usec)   : 10=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=47.16%, 20=27.36%, 50=23.52%
  lat (msec)   : 100=1.84%, 250=0.09%
   READ: bw=437MiB/s (458MB/s), 437MiB/s-437MiB/s (458MB/s-458MB/s), io=32.0GiB (34.4GB), run=75068-75068msec


RAM
Primary/Secondary 16G
Remote 256G

Network
The Primary and Secondary PBS have a 10G Uplink to the Remote. Tested with iperf.
Code: 
[SUM]   0.00-30.00  sec  32.7 GBytes  9.36 Gbits/sec  5138             sender
[SUM]   0.00-30.00  sec  32.7 GBytes  9.36 Gbits/sec                  receiver

Local Benchmarks against the datastore

PBS Primary
Code: 
Uploaded 1616 chunks in 5 seconds.
Time per request: 3099 microseconds.
TLS speed: 1353.32 MB/s
SHA256 speed: 1848.81 MB/s
Compression speed: 561.06 MB/s
Decompress speed: 870.20 MB/s
AES256/GCM speed: 10910.91 MB/s
Verify speed: 593.14 MB/s
┌───────────────────────────────────┬──────────────────────┐
│ Name                              │ Value                │
╞═══════════════════════════════════╪══════════════════════╡
│ TLS (maximal backup upload speed) │ 1353.32 MB/s (110%)  │
├───────────────────────────────────┼──────────────────────┤
│ SHA256 checksum computation speed │ 1848.81 MB/s (91%)   │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 compression speed    │ 561.06 MB/s (75%)    │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 decompression speed  │ 870.20 MB/s (73%)    │
├───────────────────────────────────┼──────────────────────┤
│ Chunk verification speed          │ 593.14 MB/s (78%)    │
├───────────────────────────────────┼──────────────────────┤
│ AES256 GCM encryption speed       │ 10910.91 MB/s (299%) │
└───────────────────────────────────┴──────────────────────┘


PBS Secondary
Code: 
Uploaded 2078 chunks in 5 seconds.
Time per request: 2407 microseconds.
TLS speed: 1741.96 MB/s
SHA256 speed: 1851.47 MB/s
Compression speed: 576.47 MB/s
Decompress speed: 899.40 MB/s
AES256/GCM speed: 11323.67 MB/s
Verify speed: 605.19 MB/s
┌───────────────────────────────────┬──────────────────────┐
│ Name                              │ Value                │
╞═══════════════════════════════════╪══════════════════════╡
│ TLS (maximal backup upload speed) │ 1741.96 MB/s (141%)  │
├───────────────────────────────────┼──────────────────────┤
│ SHA256 checksum computation speed │ 1851.47 MB/s (92%)   │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 compression speed    │ 576.47 MB/s (77%)    │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 decompression speed  │ 899.40 MB/s (75%)    │
├───────────────────────────────────┼──────────────────────┤
│ Chunk verification speed          │ 605.19 MB/s (80%)    │
├───────────────────────────────────┼──────────────────────┤
│ AES256 GCM encryption speed       │ 11323.67 MB/s (311%) │
└───────────────────────────────────┴──────────────────────┘

PBS Remote
Code: 
Uploaded 831 chunks in 5 seconds.
Time per request: 6028 microseconds.
TLS speed: 695.74 MB/s
SHA256 speed: 455.46 MB/s
Compression speed: 411.80 MB/s
Decompress speed: 653.62 MB/s
AES256/GCM speed: 2902.15 MB/s
Verify speed: 268.16 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 695.74 MB/s (56%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 455.46 MB/s (23%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 411.80 MB/s (55%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 653.62 MB/s (55%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 268.16 MB/s (35%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2902.15 MB/s (80%) │
└───────────────────────────────────┴────────────────────┘

Problem
The sync task runs very slow, giving anywhere from 1 to 10 chunks per second. Sync task is the only thing that runs, no other jobs running in that timeframe.

2026-10-19T19:18:26Z/drive-scsi0.img.fidx: downloaded 1.161 GiB (722.469 KiB/s)
2026-10-20T13:00:01Z/drive-scsi0.img.fidx: downloaded 1.247 GiB (4.457 MiB/s)
2026-10-20T19:22:03Z/drive-scsi0.img.fidx: downloaded 1.193 GiB (15.674 MiB/s)

From my understanding the sync should be faster as the backup. Since the backup has to compare which chunks he already has and which to write, but the sync already knows exactly which chunks he needs. Of course, that's only if the sync target is fast enough.

I know that the recommendation for PBS is to run SSD and way more RAM, but in the current economy we just couldn't splurge on that. We didn't expect line-speed for the sync, but we also didn't expect to see KiB/s in the log.

PS: For transparency, I also opened a ticket for this, but I wanted to hear the communities opinion on this too.
 
Code: 
I know that the recommendation for PBS is to run SSD and way more RAM, but in the current economy we just couldn't splurge on that.
PBS itself does not need much RAM. If you use ZFS then you can use spare RAM for the ARC cache but if you don't have much RAM, reduce ARC cache.

I run a virtualized PBS on a NAS (ext4 filesystem) that pulls from the primary bare metal PBS and it uses ~ 2,2GB RAM. Sure, i have a small setup but 256GB for PBS seems overkill ...
 
could you provide the full task logs? what about host metrics on both ends?
 
fabian said:
could you provide the full task logs? what about host metrics on both ends?
Click to expand...
Attached is one of the Task Logs. I can provide more, but to my eye they all look quite similar. This sync was quite large because we added a 20TB fileserver to the backup previously to this sync.

Regarding metrics. I let claude write me a script that gives back the median of 4 hour timebuckets from the PBS API Results. Since the Job ran 15 days giving every single datapoint would be quite a lot. But I can provide that if needed.
 

Attachments

which version of PBS are you running (on each of the three systems)?

with "metrics" I meant
- start a sync
- look on both ends how the CPU/memory/IO situation looks like
- check if any particular threads or processes or disks are pegged
 
fabian said:
which version of PBS are you running (on each of the three systems)?

with "metrics" I meant
- start a sync
- look on both ends how the CPU/memory/IO situation looks like
- check if any particular threads or processes or disks are pegged
Click to expand...
They all have been updated recently at the same time. All run the same version
proxmox-backup-server 4.2.8-1 running version: 4.2.8

I've attached those metrics to my last post. I've looked at a lot of those metrics, and all I can say is that the systems look idle during the slow syncs. CPU/memory/IO all low. No queues, no high latency.

I've started a new instance of the sync that took 15 days, which initial looks ok, writes 100-200mb/s on the datastore. But it currently uses 8 workers, so I assume once the smaller VMs are done and only 1-2 sessions remain it will slow down again. Then I will create new metrics.

But maybe the recent update also fixed the problems. The most recent sync from Primary to Remote finished in 38 Minutes.
Code: 
2026-10-16 06:56:19  pbs-primarybs-primary:backupbs-primary:default-hourly-sync-primary   39h 10m 45s  OK
  2026-10-17 22:07:04  pbs-primarybs-primary:backupbs-primary:default-hourly-sync-primary   44h 41m 39s  OK
  2026-10-19 18:49:10  pbs-primarybs-primary:backupbs-primary:default-hourly-sync-primary   34h 52m 20s  ERROR: sync failed with some errors.
  2026-10-21 19:33:00  pbs-primarybs-primary:backupbs-primary:default-hourly-sync-primary       38m 43s  OK

thanks!

PS: Is my assumption correct that syncs should be faster than backups (if all involved systems have exactly the same hardware)?
PPS: Ignore the "hourly" in the syncs name, we previously thought that we could set up hourly syncs of the datastore
 
Last edited:
I asked about the version because your logs contain an error that looks rather outdated ;)

your previous version likely lacked a few improvements:
- parallel syncing
- priming the cache of reusable chunks with the last local snapshot in each group

and yes, in general (non-decrypting/encrypting) syncs should be as fast or faster than the corresponding backups (all other things like storage and network being equal), depending on how your backups are created. any backup creation mode other than file-based with change-detection metadata and a valid base snapshot, or image-based with a valid dirty bitmap matching a valid base snapshot will need to read the input data to deduplicate client-side, and that part is not needed for syncing.
 
You must log in or register to reply here.
Top Bottom
Back