S3 verification - lots of failed verifications

[Str1atum]

I am using S3 storage on Hetzner Cloud on PBS 4.1.0 and experience lots of failed verifications, mostly with there types of errors:

2025-12-12T08:56:25+01:00: verify pbs-cloud-s3:vm/107/2025-11-09T15:01:08Z/drive-scsi1.img.fidx failed: chunks could not be verified
2025-12-12T08:56:25+01:00: verified 1195904.48/1351060.00 MiB in 97706.38 seconds, speed 12.24/13.83 MiB/s (62 errors)

Is this due to the beta status or is S3 object storage not as reliable per se?

Best,
Str1atum

 

[Chris]

Hi,

Is this due to the beta status or is S3 object storage not as reliable per se?

no, this is not due to s3 not being reliable per se. But check your systemd journal from the timespan of the sync job in case there is some information logged on e.g. transient networking errors which might lead to the verification failure.

Note: there was a bug in a prior version of PBS, fixed since 4.0.20, which could lead to good chunks being marked as bad. Such chunks will have been renamed on the s3 store and the local store cache, therefore leading to missing chunks. You can try to recover from that state by renaming the chunks with <digest>.0.bad to <digest> on your s3 store. By this, the next verification can find them again. If they are truely bad, they will be marked as bad again.

In any case it is recommended to verify the last backup snapshot of each group, as in case that fails verification, the next backup run is forced to re-upload chunks and not allowed to reuse chunks already known to the server. By this the bad chunks might be restored as well, if the data is still present on the source.

 

[krugson]

Yes, I am using S3 storage via Hetzner Cloud as well. I have a small cluster setup consisting of a Strato vServer, a Hetzner Dedicated Server. I’ve been running a VM on a Strato node with Proxmox Backup Server 4.1.0 on ZFS. I am using S3 storage via Hetzner Cloud as well. I frequently encounter the "verification fails" issue, but more often during the verify jobs — though this could be coincidental.

I’m encountering the same verification errors in the log via the pbs GUI as @Str1atum.


journalctl --since "2025-12-13 10:41:30" --until "2025-12-13 10:58:00"
to track my verify job, results with:

Dec 13 10:49:59 pbs proxmox-backup-proxy[584]: <?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>GatewayTimeout</Code>
<Message>The server did not respond in time.</Message>
<RequestId>N/A</RequestId>
<HostId>N/A</HostId>
</Error>

Dec 13 10:49:59 pbs proxmox-backup-proxy[584]: can't verify chunk, load failed - unexpected status code 504 Gateway Timeout

If I ping the destination of the S3 storage, latency seem ok and no obvious packet loss.

64 bytes from static.88-198-120-0.clients.your-server.de (88.198.120.0): icmp_seq=5 ttl=51 time=13.6 ms
64 bytes from static.88-198-120-0.clients.your-server.de (88.198.120.0): icmp_seq=6 ttl=51 time=13.5 ms
64 bytes from static.88-198-120-0.clients.your-server.de (88.198.120.0): icmp_seq=7 ttl=51 time=13.6 ms