Please can we have this failed chunk retry job, pretty please.

Jan 13, 2026
41
3
8
I know that the verify job tries, I think it's 3 times, to verify a chunk. We have large VM's one in particular has a disk size of 12TB and it's rare a verify job doesn't succeed without one failed chuck. The error isn't that it's corrupt it's a 500 error from the storage provider. On a 1Gb link that a 2-3 day job on the one VM. The only solution is to verify that VM's backup again so another 2-3 days and hope there isn't another 500 error. We can only run that after the complete verify job of all VMs is complete and that's 5-6 days in total. Could we get the option to have verify job or option that will verify just problem/missed chunks, like a re-verify job? This would save huge amounts of time, be done in seconds not days...

Nov 05 10:31:15 pbs proxmox-backup-proxy[2062]: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please retry the operation again later.</Message><RequestId>109C5D241600F515:B</RequestId><HostId>Z9bPvU+JpygAOHKz5GIJ4WIz1I5XQsF2cjkQX3GY4j4Jjc9fqZHB+8SNFjftj9ecF1G40Ns6gR7v</HostId><CMReferenceId>MTc5Mzg3NDMyNSBSNDAyLVU0Ni5wcm9kOC5sZDcgMTgxOTM2MzAzNzoxNzU4MzYyOToyNQ==</CMReferenceId></Error>
Nov 05 10:31:15 pbs proxmox-backup-proxy[2062]: can't verify chunk, load failed - unexpected status code 500 Internal Server Error
 
Good to see 7606 already has a patch on the dev list. Until that ships, a few settings may take some of the pain out of it.

A 500 InternalError from an S3 provider often shows up when it's under load. Newer PBS versions have limit-passive-requests on the S3 endpoint, a cap on GET and HEAD per second. Setting that a bit under what your provider allows may get rid of most of those failed chunks during verify. Also check read-threads on the verify job. The default is 1 reader. 12TB at full 1Gb line rate is about 27 hours, so if it takes 2 to 3 days, one reader waiting on S3 latency is probably part of it. Raise it a step or two together with the cap and watch whether the 500s come back.

The other thing I'd do is put that one VM in its own namespace with its own verify job (verify-job takes --ns). Then when its run fails you can repeat just that one straight away, not after the 5 to 6 day job for the whole datastore.

Disclosure: I'm founder and CEO of CyberSnap.
 
Thanks UriShay-CyberSnap I have been using 4 readers for a while to max the connection, or course it doesn't run at max all the time because of other traffic and I can see it speeds up and down during the process but of course it's the internet. I've done this testing on 2 sites at 1Gb one of which the internet is mostly idle and it's still up and down, not unusual really.

If a VM fails then I rescan that backup in isolation.

As I understand if I run a verify job, verified chunks don't get scanned twice regardless of how many VMs might be using the same chunks due to de-dupe. So it's perhaps more efficient to have all the VMs running in the same verify job. Running a VM in it's own verify job may result is chunks being scanned twice if they are used by VMs from different verify jobs.

Have I got this right?
 
Thanks UriShay-CyberSnap I have been using 4 readers for a while to max the connection, or course it doesn't run at max all the time because of other traffic and I can see it speeds up and down during the process but of course it's the internet. I've done this testing on 2 sites at 1Gb one of which the internet is mostly idle and it's still up and down, not unusual really.

If a VM fails then I rescan that backup in isolation.

As I understand if I run a verify job, verified chunks don't get scanned twice regardless of how many VMs might be using the same chunks due to de-dupe. So it's perhaps more efficient to have all the VMs running in the same verify job. Running a VM in it's own verify job may result is chunks being scanned twice if they are used by VMs from different verify jobs.

Have I got this right?

Yes, that matches my understanding. Within one verify job PBS keeps track of the chunks it already checked, so a chunk shared by several VMs is read once per run, and a separate job for that one VM would pull those shared chunks over S3 a second time. So the namespace idea only pays off if rerunning that VM fast matters more than the extra reads. Rechecking the failed backup on its own, like you already do, gets you most of that anyway.

Disclosure: I work at CyberSnap.
 
The error isn't that it's corrupt it's a 500 error from the storage provider. On a 1Gb link that a 2-3 day job on the one VM. The only solution is to verify that VM's backup again so another 2-3 days and hope there isn't another 500 error. We can only run that after the complete verify job of all VMs is complete and that's 5-6 days in total. Could we get the option to have verify job or option that will verify just problem/missed chunks, like a re-verify job?
Might have been to quick on this one, what is being worked on is improvements to the retry logic. The 500 errors status code is however already being tried up to 3 times in recent version (could you provide output of proxmox-backup-manager version --verbose), so the error shows up only after the last retry failed as well. Could you therefore also provide information about what S3 provider you are using and which region your bucket is located in? Might help to further optimize the retry logic.

Only checking chunks not recently verified requires some form of tracking, which might not be straight forward to achieve in a generic and efficient manner.