GC failing due to "can't read index" of deleted backup

[jmann]

Version is 2.1.

I noticed our PBS host started to fail on GC a while back and the volume size is slowly growing due to GC not proceeding. The failure in the logs is:

2022-11-14T00:34:05-08:00: TASK ERROR: can't read index '/mnt/backups/vm/1463/2022-10-22T07:50:31Z/drive-scsi0.img.fidx' - got unknown magic number

I found the affected VM's backup, and deleted the backup group through PBS, and re-ran the GC, and it still happens. I have no idea how to recover from this. I presume there are chunks that may have belonged to that backup, and it can't tie them to that image anymore, but I outright deleted that image.

How do I recover from this to allow GC to continue? Is there any way to get GC to correct these sorts of errors?

 

[Matthias.]

You said you deleted the group through PBS, but the file still exists?

The error message sounds to me like the image archive file got corrupted.
What's the output of hexdump -n 8 /mnt/backups/vm/1463/2022-10-22T07:50:31Z/drive-scsi0.img.fidx?

 

[jmann]

I have since cleaned up all of the affected VMs, but there were several. In this case, the complication was having to delete them by hand, and I was hoping there was a flag or something that could be passed to the garbage collector to dispose of invalid backups, primarily since the GC had to restart every time - and this is a 125T volume so the GC takes quite a while.