[SOLVED] Verify Error, "error reading a body from connection" could this be made retriable?

M

Mike_Scholes

Member
Proxmox Subscriber
Jan 13, 2026
38
3
8
Aug 05 02:32:58 pbs proxmox-backup-proxy[2125]: can't verify chunk, load failed - error reading a body from connection

Pretty sure this is a network glitch as it's a 7.9TB VM and takes around 44 hours to verify. Seems like something to add as a retriable ?

Backup Server 4.2.3
 
And then

Aug 06 13:29:13 pbs proxmox-backup-proxy[2125]: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>InternalError</Code><Message>We encountered an internal error. Please retry the operation again later.</Message><RequestId>C5F760A0EB694227:B</RequestId><HostId>VengTGyCtNaRa6AQMdKiMcTCJM1UjCZVl9WpiBZZ9fY5WVb9UEx39nzsVvKW9TYMKacB34/aOS2l</HostId><CMReferenceId>MTc4NjAxOTM1MyBSMjA0LVUyMS5wcm9kOC5sZDcgNjMxMzcxNTcwOjYwNzExNzE6ODk=</CMReferenceId></Error>
 
Mike_Scholes said:
And then

Aug 06 13:29:13 pbs proxmox-backup-proxy[2125]: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>InternalError</Code><Message>We encountered an internal error. Please retry the operation again later.</Message><RequestId>C5F760A0EB694227:B</RequestId><HostId>VengTGyCtNaRa6AQMdKiMcTCJM1UjCZVl9WpiBZZ9fY5WVb9UEx39nzsVvKW9TYMKacB34/aOS2l</HostId><CMReferenceId>MTc4NjAxOTM1MyBSMjA0LVUyMS5wcm9kOC5sZDcgNjMxMzcxNTcwOjYwNzExNzE6ODk=</CMReferenceId></Error>
Click to expand...
These are already retired up to 3 times, and since you see them logged in the systemd journal this indicates that the s3 client gave up after that.

Please try setting a request rate limit for passive requests on the api endpoint, as introduced with PBS 4.2.4 (see changelog), that should help reduce the load on you S3 provider and avoid the connection drops and unnecessary retries. You might estimate useful GET request rates from the datastore summary panel.
 
Last edited:
Thanks for that I've updated and can see the setting but can't make a lot of sense from the datastore summary panel to get a meaningful number. The panel show GET:27512781 Do you know what a meaningful limit might be on our 1GB connection?
 
Mike_Scholes said:
And now I go to wipe the GET rate, so I remove the entry and it shows unlimited, click save, go back in at it back to 10. Can't remove the setting?
Click to expand...
Can reproduce, this is a bug which is indirectly fixed by the followup patch which is currently not applied yet [0]. You may remove the setting by manually editing the config in /etc/proxmox-backup/s3.cfg instead as workaround, removing the limit-passive-requests line from the endpoint config.

Mike_Scholes said:
I tried at 2000, then 1000, then set GET requests to 10 which as you can imagine significantly cut bandwidth but still the errors occurred
Click to expand...
That's unexpected and might point to a different issue then. What provider are you using? Are you connected trough a proxy or VPN to the S3 API?

Mike_Scholes said:
Thanks for that I've updated and can see the setting but can't make a lot of sense from the datastore summary panel to get a meaningful number. The panel show GET:27512781 Do you know what a meaningful limit might be on our 1GB connection?
Click to expand...
What I intended is to estimate it based from the time dependent GET request value changes from the graph during a verification/restore, as that is what I would expect to be the upper limit.

[0] https://lore.proxmox.com/pbs-devel/20260804112937.358002-2-c.ebner@proxmox.com/
 
I'm using Wasabi and up until now has been rock solid as has Proxmox. Anyway I went to my remote systems in a different part of the country where I have a read only datastore to test restores and when I try to refresh the contents it also failed so now I'm thinking this is Wasabi. My remote system is uses a totally different ISP so any crossover with systems will be at Wasabi or very close. I will update here when I get any info.

Regardless good to know verification has retries built in.
 
You must log in or register to reply here.
Top Bottom
Back