Failed Verification after backup and syncing. How To Fix?

[Jarvar]

I have a Proxmox Backup Server in two locations A and B. One on site which makes a nightly backup. That gets synced to a remote site. The remote site gets synced offsite. However, in order to be more redundant I have also begun syncing the backups from location A to location B from the remote Proxmox Backup Server.

I am experiencing 3 failed verifications after syncing. out of 26 OK.

Is there any way to repair these?
Thank you.

 

[Dunuin]

When there is a failed chunk and you back that VM up again, and the chunks still exist, PBS will replace the bad chunks with the good ones. No idea if a sync would also replace bad chunks with good ones, but would make sense to me.
I would first run a re-verify task on the source PBS for the VMs with the failed chunks and see if they were already damaged before the sync. If they are still healthy I would run that sync job again and see if PBS will replace the bad chunks with good ones.

 

[Jarvar]

When there is a failed chunk and you back that VM up again, and the chunks still exist, PBS will replace the bad chunks with the good ones. No idea if a sync would also replace bad chunks with good ones, but would make sense to me.
I would first run a re-verify task on the source PBS for the VMs with the failed chunks and see if they were already damaged before the sync. If they are still healthy I would run that sync job again and see if PBS will replace the bad chunks with good ones.

Somehow one of the failed verify backups fixed on one PBS.
Do you know if I can delete one of the backups and resync it will fill in the missing one if I have it on another proxmox backup server?
Thank you so much.

 

[Dunuin]

Did you try if a sync will fix it without deleting backup snapshots first? Sync jobs will pull data from another PBS, and the PBS now knows what chunks are damaged, so wouldn't wonder if it then tries to pull a healthy chunk, even when you previously synced all backup snapshots already.
If it won't replace the bad chunks, this would be a great idea for a feature request.

 

[Jarvar]

Did you try if a sync will fix it without deleting backup snapshots first? Sync jobs will pull data from another PBS, and the PBS now knows what chunks are damaged, so wouldn't wonder if it then tries to pull a healthy chunk, even when you previously synced all backup snapshots already.
If it won't replace the bad chunks, this would be a great idea for a feature request.

I tried, it doesn't look like it's replacing the bad chunks. Also when I run a verify on the failed backup it just stalls for hours. I don't think it's normal. I've upgraded to the latest PBS 2.4-1 too.
Anymore thoughts? it's also weird why some have failed and others don. If I can just delete the offending backup and resync that would be the easiest. Or else I need to create a new namespace and resync over there the whole store....

 

[Dunuin]

Also when I run a verify on the failed backup it just stalls for hours.

Maybe a failing disk? Did you check the syslog and SMART for anything remarkable?
In case you are using ZFS you could also run a scrub to see if ZFS can restore that chunk.

 

[Jarvar]

Maybe a failing disk? Did you check the syslog and SMART for anything remarkable?
In case you are using ZFS you could also run a scrub to see if ZFS can restore that chunk.

I'll take a look, but I have weekly scrubs going on. I have new disks put in at one place and on a remote server. It's very unlikely a failing disk at two locations. I did a scrub on the 2nd and 4th of April without any errors.