PBS S3 backend: can the request rate be throttled to avoid 429 Too Many Requests?

[ma2kz]

Hello,

I would like to confirm whether Proxmox Backup Server currently has any configuration option to limit or throttle the request rate when using the S3 backend.

This is not intended as a feature request. I would first like to know whether such a setting already exists, or whether there is any recommended configuration on the PBS side for this situation.

I am testing backups from Proxmox Backup Server to an S3-compatible object storage provider. When backing up a VM, the backup sometimes fails with 429 Too Many Requests.

According to the object storage provider’s documentation, the cause seems clear:

• The provider enforces a limit of 100 requests per second.
• If this limit is exceeded, the service returns 429 Too Many Requests.
• The provider cannot relax this limit.
• They recommend that clients limit the number of requests per second, queue requests, or implement similar client-side handling.

A small LXC backup of around 8 GB succeeds, so I assume the object storage provider is not rejecting PBS backups in general. The issue seems to occur only when the backup generates enough chunk upload requests to exceed the provider’s request rate limit.

Environment:

root@pvenode-hv01:~# proxmox-backup-manager versions --verbose
proxmox-backup                    unknown       running kernel: 7.0.6-2-pve
proxmox-backup-server             4.2.1-1       running version: 4.2.1
proxmox-kernel-helper             9.2.0
proxmox-kernel-7.0                7.0.6-2
proxmox-kernel-7.0.6-2-pve-signed 7.0.6-2
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6
ifupdown2                         3.3.0-1+pmx12
libjs-extjs                       7.0.0-5
proxmox-backup-docs               4.2.1-1
proxmox-backup-client             4.2.1-1
proxmox-mail-forward              1.0.3
proxmox-mini-journalreader        1.6
proxmox-offline-mirror-helper     0.7.4
proxmox-widget-toolkit            5.2.3
pve-xtermjs                       6.0.0-1
smartmontools                     7.5-pve2
zfsutils-linux                    2.4.2-pve

I have attached the logs for the backup tasks on Proxmox VE and the backup server.

Thank you.

 

[Johannes S]

You can configure a put-rate-limit with the cli tool: https://pbs.proxmox.com/docs/command-syntax.html#proxmox-backup-client
Or by editing the config file see https://forum.proxmox.com/threads/s3-upload-fails-with-large-vms-cache-issue.169432/#post-792910

Sadly at the moment there doesn't seem to be a generic way to limit requests except for PUT requests, there are two bugzilla entries, maybe you want to chime in that you would like this feature?

https://bugzilla.proxmox.com/show_bug.cgi?id=6841
https://bugzilla.proxmox.com/show_bug.cgi?id=7596

 

[dw-cj]

You can configure a put-rate-limit with the cli tool: https://pbs.proxmox.com/docs/command-syntax.html#proxmox-backup-client
Or by editing the config file see https://forum.proxmox.com/threads/s3-upload-fails-with-large-vms-cache-issue.169432/#post-792910

Sadly at the moment there doesn't seem to be a generic way to limit requests except for PUT requests, there are two bugzilla entries, maybe you want to chime in that you would like this feature?

https://bugzilla.proxmox.com/show_bug.cgi?id=6841
https://bugzilla.proxmox.com/show_bug.cgi?id=7596

We use iDrive for backups and are having this issue on restores every time. For some reason, PBS just seems to fail the restore after this instead of retrying?


On the PVE node:

progress 16% (read 85899345920 bytes, zeroes = 1% (1027604480 bytes), duration 533 sec)
progress 17% (read 91268055040 bytes, zeroes = 1% (1145044992 bytes), duration 564 sec)
progress 18% (read 96636764160 bytes, zeroes = 1% (1161822208 bytes), duration 599 sec)
restore failed: unexpected status code 503 Service Unavailable

On the PBS node:

2026-06-18T11:02:24-04:00: GET /chunk
2026-06-18T11:02:24-04:00: GET /chunk
2026-06-18T11:02:24-04:00: GET /chunk
2026-06-18T11:02:24-04:00: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Resource requested is unreadable, please reduce your request rate</Message><Key>s3_idrivee2_ny_0/.chunks/de97/de97d61fd350683ffXXXXXXXc5103a3a2fe7456132335c19a2</Key><BucketName>pbs-ny</BucketName><Resource>/pbs-ny/s3_idrivee2_ny_0/.chunks/de97/de97d61fd350683ffXXXXXXXc5103a3a2fe7456132335c19a2</Resource><RequestId>18BA350FD22770C1</RequestId><HostId>XXXXXXXXX-aefa-4f6a-b050-XXXXXXXX</HostId></Error>
2026-06-18T11:02:24-04:00: GET /chunk: 400 Bad Request: unexpected status code 503 Service Unavailable
2026-06-18T11:02:24-04:00: GET /chunk
2026-06-18T11:02:24-04:00: GET /chunk
2026-06-18T11:02:24-04:00: GET /chunk
2026-06-18T11:02:24-04:00: GET /chunk
2026-06-18T11:02:24-04:00: GET /chunk
2026-06-18T11:02:24-04:00: TASK ERROR: connection error: connection reset