Hello,
I have have two PBS set up, where one syncs some snapshots from the other. The Origin PBS now has several failed snapshots which fail verification because of missing chunks. I am aware of this and know how it happened.
But the syncing PBS stops right after encountering the first failed snapshot and does not try to sync later snapshots of that group (although these are verfied good ones).
For VM 123 i have manually deleted the failed snapshots, so it continues there - but for the following VMs, which have snapshots for the exact same days as VM 123 it will stop right after coming up to the first failed one.
Can anyone reproduce this or is this even intended behaviour?
I have have two PBS set up, where one syncs some snapshots from the other. The Origin PBS now has several failed snapshots which fail verification because of missing chunks. I am aware of this and know how it happened.
But the syncing PBS stops right after encountering the first failed snapshot and does not try to sync later snapshots of that group (although these are verfied good ones).
For VM 123 i have manually deleted the failed snapshots, so it continues there - but for the following VMs, which have snapshots for the exact same days as VM 123 it will stop right after coming up to the first failed one.
Can anyone reproduce this or is this even intended behaviour?
2023-01-05T18:10:41+01:00: Starting datastore sync job 'NECKARetaSAN:ZFS:NECKAR:s-891cb411-1333'
2023-01-05T18:10:41+01:00: sync datastore 'ZFS' from 'NECKAR/PetaSAN'
2023-01-05T18:10:41+01:00: ----
2023-01-05T18:10:41+01:00: Syncing datastore 'PetaSAN', root namespace into datastore 'ZFS', namespace 'NECKAR'
2023-01-05T18:10:43+01:00: found 41 groups to sync
*SNIP*
2023-01-05T18:10:46+01:00: percentage done: 53.66% (22/41 groups)
2023-01-05T18:10:46+01:00: skipped: 39 snapshot(s) (2022-08-26T22:39:26Z .. 2023-01-03T18:55:51Z) older than the newest local snapshot
2023-01-05T18:10:46+01:00: sync snapshot vm/123/2022-12-28T21:02:56Z
2023-01-05T18:10:46+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:10:46+01:00: sync archive drive-virtio1.img.fidx
2023-01-05T18:11:10+01:00: downloaded 629425730 bytes (25.45 MiB/s)
2023-01-05T18:11:10+01:00: sync archive drive-virtio0.img.fidx
2023-01-05T18:14:18+01:00: downloaded 6094837600 bytes (30.80 MiB/s)
2023-01-05T18:14:18+01:00: got backup log file "client.log.blob"
2023-01-05T18:14:18+01:00: sync snapshot vm/123/2022-12-28T21:02:56Z done
2023-01-05T18:14:18+01:00: percentage done: 55.78% (22/41 groups, 33/38 snapshots in group #23)
2023-01-05T18:14:18+01:00: sync snapshot vm/123/2022-12-29T19:50:07Z
2023-01-05T18:14:18+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:14:18+01:00: sync archive drive-virtio1.img.fidx
2023-01-05T18:14:41+01:00: downloaded 670321376 bytes (28.31 MiB/s)
2023-01-05T18:14:41+01:00: sync archive drive-virtio0.img.fidx
2023-01-05T18:17:51+01:00: downloaded 6328921113 bytes (31.73 MiB/s)
2023-01-05T18:17:51+01:00: got backup log file "client.log.blob"
2023-01-05T18:17:51+01:00: sync snapshot vm/123/2022-12-29T19:50:07Z done
2023-01-05T18:17:51+01:00: percentage done: 55.84% (22/41 groups, 34/38 snapshots in group #23)
2023-01-05T18:17:52+01:00: sync snapshot vm/123/2022-12-30T19:05:45Z
2023-01-05T18:17:52+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:17:52+01:00: sync archive drive-virtio1.img.fidx
2023-01-05T18:18:23+01:00: downloaded 1043340284 bytes (31.91 MiB/s)
2023-01-05T18:18:23+01:00: sync archive drive-virtio0.img.fidx
2023-01-05T18:21:40+01:00: downloaded 6454772900 bytes (31.19 MiB/s)
2023-01-05T18:21:40+01:00: got backup log file "client.log.blob"
2023-01-05T18:21:40+01:00: sync snapshot vm/123/2022-12-30T19:05:45Z done
2023-01-05T18:21:40+01:00: percentage done: 55.91% (22/41 groups, 35/38 snapshots in group #23)
2023-01-05T18:21:40+01:00: sync snapshot vm/123/2023-01-02T19:44:48Z
2023-01-05T18:21:40+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:21:40+01:00: sync archive drive-virtio1.img.fidx
2023-01-05T18:22:10+01:00: downloaded 829216550 bytes (26.82 MiB/s)
2023-01-05T18:22:10+01:00: sync archive drive-virtio0.img.fidx
2023-01-05T18:25:35+01:00: downloaded 6941740928 bytes (32.25 MiB/s)
2023-01-05T18:25:35+01:00: got backup log file "client.log.blob"
2023-01-05T18:25:35+01:00: sync snapshot vm/123/2023-01-02T19:44:48Z done
2023-01-05T18:25:35+01:00: percentage done: 55.97% (22/41 groups, 36/38 snapshots in group #23)
2023-01-05T18:25:35+01:00: sync snapshot vm/123/2023-01-03T19:14:07Z
2023-01-05T18:25:35+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:25:35+01:00: sync archive drive-virtio1.img.fidx
2023-01-05T18:26:12+01:00: downloaded 1276964624 bytes (33.14 MiB/s)
2023-01-05T18:26:12+01:00: sync archive drive-virtio0.img.fidx
2023-01-05T18:29:37+01:00: downloaded 6835017138 bytes (31.90 MiB/s)
2023-01-05T18:29:37+01:00: got backup log file "client.log.blob"
2023-01-05T18:29:37+01:00: sync snapshot vm/123/2023-01-03T19:14:07Z done
2023-01-05T18:29:37+01:00: percentage done: 56.03% (22/41 groups, 37/38 snapshots in group #23)
2023-01-05T18:29:37+01:00: sync snapshot vm/123/2023-01-04T19:30:42Z
2023-01-05T18:29:37+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:29:37+01:00: sync archive drive-virtio1.img.fidx
2023-01-05T18:30:23+01:00: downloaded 1600233699 bytes (32.93 MiB/s)
2023-01-05T18:30:23+01:00: sync archive drive-virtio0.img.fidx
2023-01-05T18:33:31+01:00: downloaded 6306779098 bytes (32.06 MiB/s)
2023-01-05T18:33:31+01:00: got backup log file "client.log.blob"
2023-01-05T18:33:31+01:00: sync snapshot vm/123/2023-01-04T19:30:42Z done
2023-01-05T18:33:31+01:00: percentage done: 56.10% (23/41 groups)
2023-01-05T18:33:31+01:00: skipped: 32 snapshot(s) (2022-08-26T23:21:26Z .. 2022-12-23T19:12:28Z) older than the newest local snapshot
2023-01-05T18:33:31+01:00: re-sync snapshot vm/124/2022-12-26T19:13:10Z
2023-01-05T18:33:31+01:00: no data changes
2023-01-05T18:33:31+01:00: re-sync snapshot vm/124/2022-12-26T19:13:10Z done
2023-01-05T18:33:31+01:00: percentage done: 58.11% (23/41 groups, 33/40 snapshots in group #24)
2023-01-05T18:33:31+01:00: sync snapshot vm/124/2022-12-27T19:13:53Z
2023-01-05T18:33:31+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:33:31+01:00: sync archive drive-virtio0.img.fidx
2023-01-05T18:33:39+01:00: percentage done: 58.17% (23/41 groups, 34/40 snapshots in group #24)
2023-01-05T18:33:39+01:00: sync group vm/124 failed - reading file "/mnt/petasan-nfs/.chunks/8645/8645d15069b438871e879308588be6fd95e719d2057a40dbc6794215079dcb11" failed: No such file or directory (os error 2)
2023-01-05T18:33:39+01:00: re-sync snapshot vm/125/2022-12-26T19:40:19Z
2023-01-05T18:33:39+01:00: no data changes
2023-01-05T18:33:39+01:00: re-sync snapshot vm/125/2022-12-26T19:40:19Z done
2023-01-05T18:33:39+01:00: percentage done: 60.55% (24/41 groups, 33/40 snapshots in group #25)
2023-01-05T18:33:39+01:00: sync snapshot vm/125/2022-12-27T19:30:03Z
2023-01-05T18:33:39+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:33:39+01:00: sync archive drive-virtio1.img.fidx
2023-01-05T18:33:40+01:00: percentage done: 60.61% (24/41 groups, 34/40 snapshots in group #25)
2023-01-05T18:33:40+01:00: sync group vm/125 failed - reading file "/mnt/petasan-nfs/.chunks/6cba/6cba6bd938dcc55e34733475069f4e0044d7c06ca1fdcdd19dcca563f4f01991" failed: No such file or directory (os error 2)
2023-01-05T18:33:41+01:00: re-sync snapshot vm/126/2022-12-26T19:21:06Z
2023-01-05T18:33:41+01:00: no data changes
2023-01-05T18:33:41+01:00: re-sync snapshot vm/126/2022-12-26T19:21:06Z done
2023-01-05T18:33:41+01:00: percentage done: 62.99% (25/41 groups, 33/40 snapshots in group #26)
2023-01-05T18:33:41+01:00: sync snapshot vm/126/2022-12-27T19:22:47Z
2023-01-05T18:33:41+01:00: sync archive qemu-server.conf.blob
2023-01-05T18:33:41+01:00: sync archive drive-virtio0.img.fidx
2023-01-05T18:33:41+01:00: percentage done: 63.05% (25/41 groups, 34/40 snapshots in group #26)
2023-01-05T18:33:41+01:00: sync group vm/126 failed - reading file "/mnt/petasan-nfs/.chunks/da43/da43f7413ca903e7e6d51f04b171fc5564540f8280a725227c6a736ce7f9478a" failed: No such file or directory (os error 2)
2023-01-05T18:33:41+01:00: re-sync snapshot vm/127/2023-01-04T19:22:38Z
2023-01-05T18:33:41+01:00: re-sync snapshot vm/127/2023-01-04T19:22:38Z done
*SNIP*
2023-01-05T18:34:03+01:00: percentage done: 100.00% (41/41 groups)
2023-01-05T18:34:03+01:00: skipped: 2 snapshot(s) (2023-01-02T19:55:22Z .. 2023-01-03T18:29:04Z) older than the newest local snapshot
2023-01-05T18:34:03+01:00: Finished syncing namespace , current progress: 40 groups, 3 snapshots
2023-01-05T18:34:03+01:00: TASK ERROR: sync failed with some errors.