PBS Failing to Verify Backups That No Longer Exist

[daaave]

Hi, I have a PBS that constantly fails verification of backups stored on AWS S3. The thing is, the backups that it's failing to verify don't exist. Here is the verify log:

2026-01-23T16:56:40-08:00: Starting datastore verify job 'AWSS3:v-820ebe34-ade4'
2026-01-23T16:56:40-08:00: verify datastore AWSS3
2026-01-23T16:56:40-08:00: found 3 groups
2026-01-23T16:56:40-08:00: using 1 read and 4 verify thread(s)
2026-01-23T16:56:40-08:00: verify group AWSS3:vm/113 (1 snapshots)
2026-01-23T16:56:40-08:00: verify AWSS3:vm/113/2026-01-08T21:47:28Z
2026-01-23T16:56:40-08:00:   check qemu-server.conf.blob
2026-01-23T16:56:40-08:00:   check drive-ide0.img.fidx
2026-01-23T17:06:08-08:00:   verified 3321.21/4816.00 MiB in 567.71 seconds, speed 5.85/8.48 MiB/s (1 errors)
2026-01-23T17:06:08-08:00: verify AWSS3:vm/113/2026-01-08T21:47:28Z/drive-ide0.img.fidx failed: chunks could not be verified
2026-01-23T17:06:09-08:00: percentage done: 33.33% (1/3 groups)
2026-01-23T17:06:09-08:00: verify group AWSS3:vm/116 (7 snapshots)
2026-01-23T17:06:09-08:00: verify AWSS3:vm/116/2026-01-23T09:00:03Z
2026-01-23T17:06:09-08:00:   check qemu-server.conf.blob
2026-01-23T17:06:09-08:00:   check drive-sata0.img.fidx
2026-01-23T18:43:01-08:00:   verified 39554.91/68236.00 MiB in 5812.55 seconds, speed 6.81/11.74 MiB/s (3 errors)
2026-01-23T18:43:01-08:00: verify AWSS3:vm/116/2026-01-23T09:00:03Z/drive-sata0.img.fidx failed: chunks could not be verified
2026-01-23T18:43:02-08:00: percentage done: 38.10% (1/3 groups, 1/7 snapshots in group #2)
2026-01-23T18:43:02-08:00: verify AWSS3:vm/116/2026-01-22T09:00:02Z
2026-01-23T18:43:02-08:00:   check qemu-server.conf.blob
2026-01-23T18:43:02-08:00:   check drive-sata0.img.fidx
2026-01-23T18:58:52-08:00:   verified 4879.52/9096.00 MiB in 950.38 seconds, speed 5.13/9.57 MiB/s (1 errors)
2026-01-23T18:58:52-08:00: verify AWSS3:vm/116/2026-01-22T09:00:02Z/drive-sata0.img.fidx failed: chunks could not be verified
2026-01-23T18:58:52-08:00: percentage done: 42.86% (1/3 groups, 2/7 snapshots in group #2)
2026-01-23T18:58:52-08:00: verify AWSS3:vm/116/2026-01-21T09:00:05Z
2026-01-23T18:58:52-08:00:   check qemu-server.conf.blob
2026-01-23T18:58:52-08:00:   check drive-sata0.img.fidx
2026-01-23T19:08:06-08:00:   verified 4216.69/7748.00 MiB in 553.33 seconds, speed 7.62/14.00 MiB/s (0 errors)
2026-01-23T19:08:06-08:00: percentage done: 47.62% (1/3 groups, 3/7 snapshots in group #2)
2026-01-23T19:08:06-08:00: verify AWSS3:vm/116/2026-01-20T09:00:05Z
2026-01-23T19:08:06-08:00:   check qemu-server.conf.blob
2026-01-23T19:08:06-08:00:   check drive-sata0.img.fidx
2026-01-23T19:22:30-08:00:   verified 4466.30/8288.00 MiB in 863.68 seconds, speed 5.17/9.60 MiB/s (1 errors)
2026-01-23T19:22:30-08:00: verify AWSS3:vm/116/2026-01-20T09:00:05Z/drive-sata0.img.fidx failed: chunks could not be verified
2026-01-23T19:22:30-08:00: percentage done: 52.38% (1/3 groups, 4/7 snapshots in group #2)
2026-01-23T19:22:30-08:00: verify AWSS3:vm/116/2026-01-19T09:00:00Z
2026-01-23T19:22:30-08:00:   check qemu-server.conf.blob
2026-01-23T19:22:30-08:00:   check drive-sata0.img.fidx
2026-01-23T19:44:55-08:00:   verified 8053.00/15084.00 MiB in 1344.07 seconds, speed 5.99/11.22 MiB/s (1 errors)
2026-01-23T19:44:55-08:00: verify AWSS3:vm/116/2026-01-19T09:00:00Z/drive-sata0.img.fidx failed: chunks could not be verified
2026-01-23T19:44:55-08:00: percentage done: 57.14% (1/3 groups, 5/7 snapshots in group #2)
2026-01-23T19:44:55-08:00: verify AWSS3:vm/116/2026-01-18T09:00:08Z
2026-01-23T19:44:55-08:00:   check qemu-server.conf.blob
2026-01-23T19:44:55-08:00:   check drive-sata0.img.fidx
2026-01-23T19:53:25-08:00:   verified 3791.49/7224.00 MiB in 509.73 seconds, speed 7.44/14.17 MiB/s (0 errors)
2026-01-23T19:53:25-08:00: percentage done: 61.90% (1/3 groups, 6/7 snapshots in group #2)
2026-01-23T19:53:25-08:00: verify AWSS3:vm/116/2026-01-17T09:00:02Z
2026-01-23T19:53:25-08:00:   check qemu-server.conf.blob
2026-01-23T19:53:25-08:00:   check drive-sata0.img.fidx
2026-01-23T20:02:00-08:00:   verified 3827.90/7144.00 MiB in 515.01 seconds, speed 7.43/13.87 MiB/s (0 errors)
2026-01-23T20:02:01-08:00: percentage done: 66.67% (2/3 groups)
2026-01-23T20:02:01-08:00: verify group AWSS3:vm/117 (7 snapshots)
2026-01-23T20:02:01-08:00: verify AWSS3:vm/117/2026-01-23T09:00:10Z
2026-01-23T20:02:01-08:00:   check qemu-server.conf.blob
2026-01-23T20:02:01-08:00:   check drive-sata1.img.fidx
2026-01-23T23:09:10-08:00:   verified 50417.62/123740.00 MiB in 11228.88 seconds, speed 4.49/11.02 MiB/s (3 errors)
2026-01-23T23:09:10-08:00: verify AWSS3:vm/117/2026-01-23T09:00:10Z/drive-sata1.img.fidx failed: chunks could not be verified
2026-01-23T23:09:10-08:00: percentage done: 71.43% (2/3 groups, 1/7 snapshots in group #3)
2026-01-23T23:09:10-08:00: verify AWSS3:vm/117/2026-01-22T09:00:01Z
2026-01-23T23:09:10-08:00:   check qemu-server.conf.blob
2026-01-23T23:09:10-08:00:   check drive-sata1.img.fidx
2026-01-23T23:42:35-08:00: TASK ERROR: verification failed - job aborted

Notice all the dates are from back in January? We don't retain backups on S3 for that long. Some of those VMs have long since been deleted and removed from the backup job.

And here is a list of the current backups on the AWSS3 storage.



How can I tell PBS to stop trying to verify jobs from months ago that are no longer existing?

Note: I have been using the S3 backup feature since it was first released. Should I just start fresh with a new bucket? That would be possible but obviously not ideal.

Thanks!

 

[Chris]

How can I tell PBS to stop trying to verify jobs from months ago that are no longer existing?

Have you checked that the snapshots are indeed no longer present on the local datastore cache on the PBS side as well as on the S3 object store? Do you maybe have object versioning or other retention settings enabled on the AWS side which would keep the index files from being cleaned up?

Note: I have been using the S3 backup feature since it was first released. Should I just start fresh with a new bucket? That would be possible but obviously not ideal.

If you no longer have the snapshots on the S3 object store, run a s3-refresh to resync the datastore contents from the object store to the PBS local store cache.

 

[daaave]

Ah, silly me - look at the dates on the left of the log file - the verify job hasn't been run since January. It was disabled 'cause it was failing. I ran a refresh and I'm trying the verify job again now that I'm running v4.2.2. Thanks!