403 Error During Backup/Restore – Rate Limit Issue with Cloud Provider

[z1ro1]

I am currently experiencing a persistent 403 error when performing backup and restore operations with Proxmox Backup Server (PBS). After some investigation, it turned out that this issue is related to the rate limit imposed by my cloud storage provider (ArvanCloud).


According to ArvanCloud’s documentation and support team, they enforce a strict limit of 100 requests per 3 seconds, which unfortunately cannot be changed on their side. When PBS performs backups/restores, it generates requests at a higher rate, which triggers the rate limiter, resulting in HTTP 403 Forbidden responses.

My environment:​

• Proxmox Backup Server: Latest version (up to date)
• Backend storage: ArvanCloud Object Storage (S3-compatible)
• Backup operation: VM backups (~several GBs each), multiple chunks per operation

What I have tried so far:​

• I attempted to use PUT rate-limit configuration on PBS (set put-rate-limit = 1), but unfortunately it did not resolve the issue. Requests are still being sent too aggressively and exceed ArvanCloud’s rate threshold.
• I verified that the problem is not bandwidth-related, but rather purely due to request-per-second limits.

My Question:​

Is there any way in Proxmox Backup Server to control or limit the number of outgoing requests per second (globally, not just per PUT), so that PBS can stay below the provider’s threshold (≤ 100 requests / 3s)?


This would be critical for anyone using PBS with providers like ArvanCloud or other S3-compatible backends with strict API request limits. Without this control, backup/restore operations are not reliable and always fail with 403 errors.


Any suggestions, configuration options, or patches would be greatly appreciated.


Thank you!

 

[UdoB]

The PBS is local and just the storage is remote in that slow cloud?

I would suspect that having a working and usable PBS with just a handful of IOPS is... impossible.

Remember that lokal HDDs with ~150 IOPS are considered too slow (for large/professional use cases) and SSD storage (with some thousand IOPS) is recommended...

Disclaimer: not tested...

Edit: if you really want to use that cloud storage try to switch from PBS to vzdump-based "classic" backup.

 

[z1ro1]

The PBS is local and just the storage is remote in that slow cloud?

I would suspect that having a working and usable PBS with just a handful of IOPS is... impossible.

Remember that lokal HDDs with ~150 IOPS are considered too slow (for large/professional use cases) and SSD storage (with some thousand IOPS) is recommended...

Disclaimer: not tested...

Edit: if you really want to use that cloud storage try to switch from PBS to vzdump-based "classic" backup.

Thanks for your reply. I think there is a small misunderstanding here.


My setup is as follows:

• PBS server is running locally with HDD storage (so IOPS is not the bottleneck).
• The problem occurs only because the remote S3-compatible backend (ArvanCloud Object Storage) enforces a hard rate limit of 100 requests per 3 seconds.

When PBS performs backup/restore, it sends requests at a higher rate than this threshold, which results in 403 Forbidden responses. This is not related to local disk performance or bandwidth, but purely to the API request-per-second limitation enforced by the storage provider.


That’s why I am asking if there is any way to configure PBS to globally throttle outgoing requests (not just PUT operations), so that it stays below the provider’s threshold.


Switching to vzdump + manual upload would bypass the problem, but it also means losing all the benefits of PBS (deduplication, chunking, efficient restores, etc.), which I’d really like to keep.

 

[Chris]

That’s why I am asking if there is any way to configure PBS to globally throttle outgoing requests (not just PUT operations), so that it stays below the provider’s threshold.

This is not implemented, request rate limits for the s3 client can currently only be set for put requests. Please open a feature request describing your usecase on https://bugzilla.proxmox.com, thanks!