Verification "error" 500 Internal Server Error on S3 datastore

[Mike_Scholes]

Hi all.

During some large VM (1.5TB pure data, 3TB VM) verify jobs I am seeing this more often than not...

Mar 06 10:42:20 pbs proxmox-backup-proxy[1989155]: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InternalError</Code><Message>We encountered an internal error. Please retry the operation again later.</Message><RequestId>F0D8F57082ADB500:B</RequestId><HostId>w+X+/wgFM0iOkVAesuwh43iDK0Bg1mFLWoW2CHw47sWtCeoYgib6a1pV30kbRbTsWznv1VNpT6Xo</HostId><CMReferenceId>MTc3Mjc5MzQzMCBSNDAyLVU0Ni5wcm9kOC5sZDcgNTYxNDIwMDQ1OjUzODg4MTQ6ODc=</CMReferenceId></Error>
Mar 06 10:42:20 pbs proxmox-backup-proxy[1989155]: can't verify chunk, load failed - unexpected status code 500 Internal Server Error

This is accessing a Wasabi bucket. I'm pretty sure the data is fine as the same data has verified previously, VM restored successfully and sync'd back to the local backup just fine. Seems this is an glitch accessing the single chunk but it does leave me with VMs with failed verification. I've tried from 2 different locations with 1Gb and 500Mb broadband respectively. Are there any plans to deal with this glitch and more volatile internet based storage as in retries etc?

 

[Chris]

A bugfix to handle such retryable errors has recently been applied in our git repository [0], it has not been packaged yet.

[0] https://git.proxmox.com/?p=proxmox.git;a=commit;h=ef4b0e26435fdb8ab7cadb49034980204f960845

 

[Mike_Scholes]

Great thanks

 

[Ericmann]

This looks like a transient server-side error rather than a data integrity issue - your successful restores confirm the data is fine. Until retry logic is added for transient errors during verification, the best workaround is to re-run the verify job manually after a short delay. Worth raising an official feature request for automatic retries on volatile remote storage.