Please can we have this failed chunk retry job, pretty please.

M

Mike_Scholes

Member
Proxmox Subscriber
Jan 13, 2026
40
3
8
I know that the verify job tries, I think it's 3 times, to verify a chunk. We have large VM's one in particular has a disk size of 12TB and it's rare a verify job doesn't succeed without one failed chuck. The error isn't that it's corrupt it's a 500 error from the storage provider. On a 1Gb link that a 2-3 day job on the one VM. The only solution is to verify that VM's backup again so another 2-3 days and hope there isn't another 500 error. We can only run that after the complete verify job of all VMs is complete and that's 5-6 days in total. Could we get the option to have verify job or option that will verify just problem/missed chunks, like a re-verify job? This would save huge amounts of time, be done in seconds not days...

Nov 05 10:31:15 pbs proxmox-backup-proxy[2062]: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please retry the operation again later.</Message><RequestId>109C5D241600F515:B</RequestId><HostId>Z9bPvU+JpygAOHKz5GIJ4WIz1I5XQsF2cjkQX3GY4j4Jjc9fqZHB+8SNFjftj9ecF1G40Ns6gR7v</HostId><CMReferenceId>MTc5Mzg3NDMyNSBSNDAyLVU0Ni5wcm9kOC5sZDcgMTgxOTM2MzAzNzoxNzU4MzYyOToyNQ==</CMReferenceId></Error>
Nov 05 10:31:15 pbs proxmox-backup-proxy[2062]: can't verify chunk, load failed - unexpected status code 500 Internal Server Error
 
Good to see 7606 already has a patch on the dev list. Until that ships, a few settings may take some of the pain out of it.

A 500 InternalError from an S3 provider often shows up when it's under load. Newer PBS versions have limit-passive-requests on the S3 endpoint, a cap on GET and HEAD per second. Setting that a bit under what your provider allows may get rid of most of those failed chunks during verify. Also check read-threads on the verify job. The default is 1 reader. 12TB at full 1Gb line rate is about 27 hours, so if it takes 2 to 3 days, one reader waiting on S3 latency is probably part of it. Raise it a step or two together with the cap and watch whether the 500s come back.

The other thing I'd do is put that one VM in its own namespace with its own verify job (verify-job takes --ns). Then when its run fails you can repeat just that one straight away, not after the 5 to 6 day job for the whole datastore.

Disclosure: I'm founder and CEO of CyberSnap.
 
You must log in or register to reply here.
Top Bottom
Back