Full datastore: Errors after 2nd GC even though all chunks were restored

[Onslow]

That must be it. Which, I guess, means there's nothing I can do to revert this.

Agreed.

If that assumption is true, I guess I am lucky.

At this point, is there even anything I can do to get into a clean state again or will I keep seeing errors when running GC and Verify?

There's some chance that all (or some) missing chunks will be recreated by the next backup.
So after all backup jobs cycle you can manually issue Verify for the backups marked as faulty. If missing chunks have been recreated, the backups will be marked as OK. If not, they will be incomplete (though not worthless, as I have already pointed out). You will not be alarmed every day because most likely your verification schedule doesn't verify all backups every day, just the newly created ones and then periodically (e.g. monthly).
In case you don't need the incomplete backups, you can delete them (you will not lose chunks used by other backups) and the following GC will remove them and errors will stop.

I think that also GC will stop showing errors after it have cleaned up the situation.

 

[Robert Obkircher]

My experience shows that corrupted chunks are renamed ...number.bad (where number is 0, 1 etc when there are more than one bad chunk of expected checksum).

So maybe these *.bad files have already been deleted by the following GC.

If no good version exists, the GC keeps the bad ones around as long as they are referenced.

It is more likely that the sweep phase simply deleted the chunks that were moved back, because mv doesn't update atime.

At this point, is there even anything I can do to get into a clean state again or will I keep seeing errors when running GC and Verify?

One more thing you could try is to scan your disks for the missing filenames with a data recovery tool. If there is not a lot of IO on your root disk the data might still be there.

 

[vtr]

Thanks @Robert Obkircher. But now I'm a bit confused. Was the issue that I moved the chunks back during GC or that atime was different?

Also, if I were in the same situation again (and I hope I won't be), what would the correct approach be? Manipulating atime before moving the chunks back?

 

[Onslow]

But now I'm a bit confused. Was the issue that I moved the chunks back during GC or that atime was different?

As I wrote in this thread a few times and Robert did as well (and this is described in the doc which I linked to for you):
most likely GC deleted the chunks because you moved them back when GC was running.
So the atime was old (not updated by GC).

Also, if I were in the same situation again (and I hope I won't be), what would the correct approach be? Manipulating atime before moving the chunks back?

No.
Just not moving the chunks back before GC finished.

 

[vtr]

Thanks again @Onslow. I'm currently putting out multiple fires at the same time, I appreciate your patience.