Sync: local rate limit

E

et1000

Member
Jan 11, 2024
16
1
8
France
Hello,

I am facing a performance issue with a local Sync Job that is completely overloading my storage. Here is the architecture of the setup (PBS is virtualized):
  • Source datastore: an S3 bucket with its local cache residing on a local SATA SSD (rpool/data).
  • Destination datastore: a local datastore running on a separate NVMe pool (pool-vm, ZFS zvol).
When the job runs, the SATA SSD gets hammered with simultaneous reads and writes, spiking the host's iowait.

Code: 
$ iostat -x 1
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          39.28    0.00   20.51   37.28    0.00    2.93

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1          8.00   2048.00     0.00   0.00    2.25   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.02   0.40
nvme1n1         16.00   4096.00     0.00   0.00    0.75   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   0.40
sda            483.00   5924.00     2.00   0.41   12.18    12.27  276.00  31920.00     0.00   0.00    7.20   115.65    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    7.87  98.10
[...]
zd192            0.00      0.00     0.00   0.00    0.00     0.00 1482.00   5928.00     0.00   0.00   12.07     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   17.88 100.10

The source SATA SSD (sda) hosting the S3 cache is at 98% utilization, constantly writing the incoming S3 stream (~32MB/s) while being read at the same time to feed the destination NVMe zvol (zd192).
I have tried multiple ways to throttle this write overhead:
  • In the Web GUI, the rate limit field for this Sync Job is grayed out and forced to "unlimited" because it is a local job.
  • I tried updating the sync job via CLI: proxmox-backup-manager sync-job update s-2d5ef083-e683 --rate-in 20MiB but this parameter seems to have no effect on local sync tasks.
  • I configured a global Traffic Control rule, but as per the documentation, local sync jobs bypass Traffic Control entirely (the rule bandwidth counter stays at 0%). "Sync jobs on the server are not affected by the configured rate limits."
Since the incoming network stream from S3 cannot be throttled inside PBS for local syncs, is there any hidden parameter, config tweak, or upcoming feature to prevent a local sync from completely choking the underlying cache disk?

Thanks!
Emile
 
You must log in or register to reply here.
Top Bottom
Back