S3 bucket verification tricks

P

p-user

Member
Jan 26, 2024
84
5
13
I seem to have problems the last week with my S3 datastore (Tuxis DaDup), which had work flawlesly since last year October.

Are there any verification tricks on the S3 bucket and/or S3 datastore?

Currently I'm running a verification job on that namespace on the S3 datastore, with the Skip verified option disabled.

It's not a mission critical system, but I don't like errors.

Are there any procedures on verifying an S3 datastore?

Any help is appreciated,

Regards,
Albert
 
Here it is. Looks like a blob that was not synchronized properly.

Workflow, I make backups once a week on a local attacfhed store, and run a sync job to the S3 storage every night.
It has run fine until last week. It's not a problem to remove the backups on the S3 storage completely and a new full sync.
But I have to remove them one at a time, is there a quicker way.



2026-06-29T02:00:00+02:00: Starting datastore verify job 'daDup:v-17e17d3c-e62a'
2026-06-29T02:00:00+02:00: task triggered by schedule '02:00'
2026-06-29T02:00:00+02:00: verify datastore daDup
2026-06-29T02:00:00+02:00: found 20 groups
2026-06-29T02:00:00+02:00: using 1 read and 4 verify thread(s)
2026-06-29T02:00:00+02:00: verify group daDup:vm/100 (1 snapshots)
2026-06-29T02:00:00+02:00: verify daDup:vm/100/2026-06-28T07:48:23Z
2026-06-29T02:00:00+02:00: check qemu-server.conf.blob
2026-06-29T02:00:00+02:00: check drive-scsi0.img.fidx
2026-06-29T02:27:24+02:00: verified 16012.45/27080.00 MiB in 1644.20 seconds, speed 9.74/16.47 MiB/s (1 errors)
2026-06-29T02:27:24+02:00: verify daDup:vm/100/2026-06-28T07:48:23Z/drive-scsi0.img.fidx failed: chunks could not be verified
2026-06-29T02:27:24+02:00: percentage done: 5.00% (1/20 groups)
2026-06-29T02:27:24+02:00: verify group daDup:vm/101 (1 snapshots)
2026-06-29T02:27:24+02:00: verify daDup:vm/101/2026-06-28T07:50:29Z
2026-06-29T02:27:24+02:00: check qemu-server.conf.blob
2026-06-29T02:27:24+02:00: check drive-scsi0.img.fidx
2026-06-29T02:39:58+02:00: verified 7620.74/14672.00 MiB in 753.53 seconds, speed 10.11/19.47 MiB/s (0 errors)
2026-06-29T02:39:58+02:00: percentage done: 10.00% (2/20 groups)
2026-06-29T02:39:58+02:00: verify group daDup:vm/102 (1 snapshots)
2026-06-29T02:39:58+02:00: verify daDup:vm/102/2026-06-28T07:49:19Z
2026-06-29T02:39:58+02:00: check qemu-server.conf.blob
2026-06-29T02:39:58+02:00: check drive-sata0.img.fidx
2026-06-29T02:43:50+02:00: verified 1863.52/6336.00 MiB in 232.21 seconds, speed 8.03/27.29 MiB/s (0 errors)
2026-06-29T02:43:50+02:00: percentage done: 15.00% (3/20 groups)
2026-06-29T02:43:50+02:00: verify group daDup:vm/103 (1 snapshots)
2026-06-29T02:43:50+02:00: verify daDup:vm/103/2026-06-28T07:52:52Z
2026-06-29T02:43:50+02:00: check qemu-server.conf.blob
2026-06-29T02:43:50+02:00: check drive-tpmstate0-backup.img.fidx
2026-06-29T02:43:51+02:00: verified 0.01/3.95 MiB in 0.05 seconds, speed 0.18/83.58 MiB/s (0 errors)
2026-06-29T02:43:51+02:00: check drive-scsi0.img.fidx
2026-06-29T03:22:53+02:00: verified 24108.04/48004.00 MiB in 2342.83 seconds, speed 10.29/20.49 MiB/s (0 errors)
2026-06-29T03:22:53+02:00: check drive-efidisk0.img.fidx
2026-06-29T03:22:53+02:00: verified 0.03/0.52 MiB in 0.01 seconds, speed 1.79/35.08 MiB/s (0 errors)
2026-06-29T03:22:54+02:00: percentage done: 20.00% (4/20 groups)
2026-06-29T03:22:54+02:00: verify group daDup:vm/104 (2 snapshots)
2026-06-29T03:22:54+02:00: verify daDup:vm/104/2026-06-28T07:57:37Z
2026-06-29T03:22:54+02:00: check qemu-server.conf.blob
2026-06-29T03:22:54+02:00: check drive-scsi0.img.fidx
2026-06-29T03:29:02+02:00: verified 3157.78/7360.00 MiB in 368.04 seconds, speed 8.58/20.00 MiB/s (0 errors)
2026-06-29T03:29:02+02:00: percentage done: 22.50% (4/20 groups, 1/2 snapshots in group #5)
2026-06-29T03:29:02+02:00: verify daDup:vm/104/2026-05-27T11:19:26Z
2026-06-29T03:29:02+02:00: check qemu-server.conf.blob
2026-06-29T03:29:02+02:00: check drive-scsi0.img.fidx
2026-06-29T03:41:21+02:00: verified 7512.36/11548.00 MiB in 738.63 seconds, speed 10.17/15.63 MiB/s (0 errors)
2026-06-29T03:41:22+02:00: percentage done: 25.00% (5/20 groups)
2026-06-29T03:41:22+02:00: verify group daDup:vm/105 (1 snapshots)
2026-06-29T03:41:22+02:00: verify daDup:vm/105/2026-06-28T07:50:26Z
2026-06-29T03:41:22+02:00: check qemu-server.conf.blob
2026-06-29T03:41:22+02:00: check drive-scsi0.img.fidx
2026-06-29T03:47:34+02:00: verified 3902.28/5712.00 MiB in 372.36 seconds, speed 10.48/15.34 MiB/s (0 errors)
2026-06-29T03:47:34+02:00: percentage done: 30.00% (6/20 groups)
2026-06-29T03:47:34+02:00: verify group daDup:vm/106 (1 snapshots)
2026-06-29T03:47:34+02:00: verify daDup:vm/106/2026-06-28T07:58:25Z
2026-06-29T03:47:34+02:00: check qemu-server.conf.blob
2026-06-29T03:47:34+02:00: check drive-ide0.img.fidx
2026-06-29T04:09:41+02:00: verified 14640.90/17936.00 MiB in 1326.49 seconds, speed 11.04/13.52 MiB/s (0 errors)
2026-06-29T04:09:41+02:00: percentage done: 35.00% (7/20 groups)
2026-06-29T04:09:41+02:00: verify group daDup:vm/107 (1 snapshots)
2026-06-29T04:09:41+02:00: verify daDup:vm/107/2026-06-28T08:00:09Z
2026-06-29T04:09:41+02:00: check qemu-server.conf.blob
2026-06-29T04:09:41+02:00: check drive-ide0.img.fidx
2026-06-29T04:11:06+02:00: verified 861.39/1416.00 MiB in 84.58 seconds, speed 10.18/16.74 MiB/s (0 errors)
2026-06-29T04:11:06+02:00: percentage done: 40.00% (8/20 groups)
2026-06-29T04:11:06+02:00: verify group daDup:vm/108 (1 snapshots)
2026-06-29T04:11:06+02:00: verify daDup:vm/108/2026-06-28T08:00:34Z
2026-06-29T04:11:06+02:00: check qemu-server.conf.blob
2026-06-29T04:11:06+02:00: check drive-scsi0.img.fidx
2026-06-29T04:16:28+02:00: verified 2439.34/8584.00 MiB in 321.84 seconds, speed 7.58/26.67 MiB/s (0 errors)
2026-06-29T04:16:28+02:00: percentage done: 45.00% (9/20 groups)
2026-06-29T04:16:28+02:00: verify group daDup:vm/109 (1 snapshots)
2026-06-29T04:16:28+02:00: verify daDup:vm/109/2026-06-28T07:52:16Z
2026-06-29T04:16:28+02:00: check qemu-server.conf.blob
2026-06-29T04:16:28+02:00: check drive-scsi0.img.fidx
2026-06-29T04:45:19+02:00: verified 19368.58/24760.00 MiB in 1731.22 seconds, speed 11.19/14.30 MiB/s (0 errors)
2026-06-29T04:45:20+02:00: percentage done: 50.00% (10/20 groups)
2026-06-29T04:45:20+02:00: verify group daDup:vm/110 (1 snapshots)
2026-06-29T04:45:20+02:00: verify daDup:vm/110/2026-06-28T08:01:35Z
2026-06-29T04:45:20+02:00: check qemu-server.conf.blob
2026-06-29T04:45:20+02:00: check drive-tpmstate0-backup.img.fidx
2026-06-29T04:45:20+02:00: verified 0.00/4.00 MiB in 0.04 seconds, speed 0.09/91.27 MiB/s (0 errors)
2026-06-29T04:45:20+02:00: check drive-scsi0.img.fidx
2026-06-29T04:51:33+02:00: verified 3897.92/6708.00 MiB in 372.85 seconds, speed 10.45/17.99 MiB/s (0 errors)
2026-06-29T04:51:33+02:00: percentage done: 55.00% (11/20 groups)
2026-06-29T04:51:33+02:00: verify group daDup:vm/111 (1 snapshots)
2026-06-29T04:51:33+02:00: verify daDup:vm/111/2026-06-28T07:56:42Z
2026-06-29T04:51:33+02:00: check qemu-server.conf.blob
2026-06-29T04:51:33+02:00: check drive-scsi0.img.fidx
2026-06-29T04:56:57+02:00: verified 2748.76/7616.00 MiB in 323.63 seconds, speed 8.49/23.53 MiB/s (0 errors)
2026-06-29T04:56:57+02:00: percentage done: 60.00% (12/20 groups)
2026-06-29T04:56:57+02:00: verify group daDup:vm/113 (1 snapshots)
2026-06-29T04:56:57+02:00: verify daDup:vm/113/2026-06-28T07:58:19Z
2026-06-29T04:56:57+02:00: check qemu-server.conf.blob
2026-06-29T04:56:57+02:00: check drive-scsi0.img.fidx
2026-06-29T05:06:19+02:00: verified 5051.68/12432.00 MiB in 561.80 seconds, speed 8.99/22.13 MiB/s (0 errors)
2026-06-29T05:06:20+02:00: percentage done: 65.00% (13/20 groups)
2026-06-29T05:06:20+02:00: verify group daDup:vm/114 (1 snapshots)
2026-06-29T05:06:20+02:00: verify daDup:vm/114/2026-06-28T19:28:57Z
2026-06-29T05:06:20+02:00: check qemu-server.conf.blob
2026-06-29T05:06:20+02:00: check drive-scsi0.img.fidx
2026-06-29T05:17:39+02:00: verified 5426.70/17168.00 MiB in 678.72 seconds, speed 8.00/25.29 MiB/s (0 errors)
2026-06-29T05:17:39+02:00: percentage done: 70.00% (14/20 groups)
2026-06-29T05:17:39+02:00: verify group daDup:vm/118 (1 snapshots)
2026-06-29T05:17:39+02:00: verify daDup:vm/118/2026-06-28T08:07:56Z
2026-06-29T05:17:39+02:00: check qemu-server.conf.blob
2026-06-29T05:17:39+02:00: check drive-scsi0.img.fidx
2026-06-29T05:35:02+02:00: verified 10701.20/19208.00 MiB in 1043.12 seconds, speed 10.26/18.41 MiB/s (0 errors)
2026-06-29T05:35:02+02:00: percentage done: 75.00% (15/20 groups)
2026-06-29T05:35:02+02:00: verify group daDup:vm/119 (2 snapshots)
2026-06-29T05:35:02+02:00: verify daDup:vm/119/2026-06-28T08:03:25Z
2026-06-29T05:35:02+02:00: check qemu-server.conf.blob
2026-06-29T05:35:02+02:00: check drive-scsi0.img.fidx
2026-06-29T05:52:56+02:00: verified 10467.32/23376.00 MiB in 1073.92 seconds, speed 9.75/21.77 MiB/s (0 errors)
2026-06-29T05:52:56+02:00: percentage done: 77.50% (15/20 groups, 1/2 snapshots in group #16)
2026-06-29T05:52:56+02:00: verify daDup:vm/119/2025-11-15T23:56:58Z
2026-06-29T05:52:56+02:00: check qemu-server.conf.blob
2026-06-29T05:52:56+02:00: check drive-scsi0.img.fidx
2026-06-29T06:05:43+02:00: verified 6896.52/12984.00 MiB in 766.29 seconds, speed 9.00/16.94 MiB/s (0 errors)
2026-06-29T06:05:43+02:00: percentage done: 80.00% (16/20 groups)
2026-06-29T06:05:43+02:00: verify group daDup:vm/120 (1 snapshots)
2026-06-29T06:05:43+02:00: verify daDup:vm/120/2026-06-28T08:07:56Z
2026-06-29T06:05:43+02:00: check qemu-server.conf.blob
2026-06-29T06:05:43+02:00: check drive-scsi0.img.fidx
2026-06-29T06:18:02+02:00: verified 6912.68/12712.00 MiB in 738.41 seconds, speed 9.36/17.22 MiB/s (0 errors)
2026-06-29T06:18:03+02:00: percentage done: 85.00% (17/20 groups)
2026-06-29T06:18:03+02:00: verify group daDup:vm/121 (1 snapshots)
2026-06-29T06:18:03+02:00: verify daDup:vm/121/2026-06-28T08:12:06Z
2026-06-29T06:18:03+02:00: check qemu-server.conf.blob
2026-06-29T06:18:03+02:00: check drive-scsi0.img.fidx
2026-06-29T06:29:13+02:00: verified 6331.47/10992.00 MiB in 670.42 seconds, speed 9.44/16.40 MiB/s (0 errors)
2026-06-29T06:29:14+02:00: percentage done: 90.00% (18/20 groups)
2026-06-29T06:29:14+02:00: verify group daDup:vm/125 (1 snapshots)
2026-06-29T06:29:14+02:00: verify daDup:vm/125/2025-12-28T15:21:21Z
2026-06-29T06:29:14+02:00: check qemu-server.conf.blob
2026-06-29T06:29:14+02:00: check drive-ide0.img.fidx
2026-06-29T06:44:50+02:00: verified 10095.95/13960.00 MiB in 935.80 seconds, speed 10.79/14.92 MiB/s (0 errors)
2026-06-29T06:44:50+02:00: percentage done: 95.00% (19/20 groups)
2026-06-29T06:44:50+02:00: verify group daDup:vm/600 (1 snapshots)
2026-06-29T06:44:50+02:00: verify daDup:vm/600/2026-06-28T08:14:10Z
2026-06-29T06:44:50+02:00: check qemu-server.conf.blob
2026-06-29T06:44:50+02:00: check drive-scsi0.img.fidx
2026-06-29T07:04:56+02:00: verified 12003.51/21840.00 MiB in 1205.59 seconds, speed 9.96/18.12 MiB/s (0 errors)
2026-06-29T07:04:56+02:00: percentage done: 100.00% (20/20 groups)
2026-06-29T07:04:56+02:00: Failed to verify the following snapshots/groups:
2026-06-29T07:04:56+02:00: ns/Backup/vm/100/2026-06-28T07:48:23Z
2026-06-29T07:04:56+02:00: TASK ERROR: verification failed - please check the log for details
Click to expand...
 
p-user said:
It's not a problem to remove the backups on the S3 storage completely and a new full sync.
Click to expand...
What would you like to achieve by that? I would rather recommend to either do pruning via remove vanished or via a dedicated prune job on the s3 store.

So from the logs it seems that only one snapshot has issues, namely ns/Backup/vm/100/2026-06-28T07:48:23Z. Is this snapshot marked as corrupt after the verify job? Are there further errors in the systemd journal around the time of the verify job?

If the snapshot is marked as corrupt, you might want to clear and resync (if push) or resync with resync corrupt flag set (if pull).
 
I run a Verify Job every day, with re-verify after 30 days. A Prune Job at 1h45, and a garbage collect at 2h00 every day. A pull sync job at 1h00 with the remove vanished and resync corrupt options ON.
 
You must log in or register to reply here.
Top Bottom
Back