PBS issue: Stat known chunks failed

H

Hurara

New Member
Jan 29, 2025
5
1
3
After upgrading Proxmox Backup Server (PBS) to version 3.3.2, several backup jobs are failing with the error:
No such file or directory (os error 2)
The error occurs during the backup process, specifically when attempting to verify or locate known chunks.

Error Details:
Sample error log:
Backup of VM xxx failed - backup finish failed: command error: stat known chunks failed - stat failed on 99bbe5de74e64812464c8ea8443969123e416f6cdc68b8d4573dfsa313983363df


Can you please confirm if there is a fix or any work around for this issue?
 
Hi,
Hurara said:
Can you please confirm if there is a fix or any work around for this issue?
Click to expand...
this is not a bug, but rather the check introduced to detect known chunks which have been reused for the backup snapshot not being present in the datastore anymore (might that they have been marked as corrupt by a verification).

This will set the verify state for the previous backup snapshot to failed, the next backup run will therefore perform a full backup, re-uploading all chunks.

I urge you to run a full verify job to check all of your snapshots, some of them are seemingly corrupt!

On a side note: This check has since been reverted, because it did not scale well for large backups, see https://forum.proxmox.com/threads/pbs-3-3-1-backup-tasks-hang-after-concluding-uploads.158812/
 
  • Like
Reactions: Johannes S
Johannes S said:
Will there be another Implementation for this chrck? Im theory it seems like a good idea to me to avoid having corrupt backups
Click to expand...
Yes, there are plans to handle this differently, e.g. https://lore.proxmox.com/pbs-devel/1736772071.mcbrxnbmft.astroid@yuna.none/

Please note however, that you already can detect this by enabling the verify new snapshots option in the datastores Options tab. That is even more thorough than the simple stat check, as it also verifies the chunk content.

So the functionality to detect these kind of corruption right away is already there!
 
  • Like
Reactions: Johannes S
Chris said:
Hi,

this is not a bug, but rather the check introduced to detect known chunks which have been reused for the backup snapshot not being present in the datastore anymore (might that they have been marked as corrupt by a verification).

This will set the verify state for the previous backup snapshot to failed, the next backup run will therefore perform a full backup, re-uploading all chunks.

I urge you to run a full verify job to check all of your snapshots, some of them are seemingly corrupt!

On a side note: This check has since been reverted, because it did not scale well for large backups, see https://forum.proxmox.com/threads/pbs-3-3-1-backup-tasks-hang-after-concluding-uploads.158812/
Click to expand...
Thanks for helping out, I do have some questions, so will it fix the issue if I delete all the failed marked snapshots and then run the backup again, and also if the stat check functionality has been reverted why I am receiving these errors, I have the latest version installed 3.3.2? Also can you please let me know how can i do the full verify job for snapshots.
 
Last edited:
Hurara said:
Thanks for helping out, I do have some questions, so will it fix the issue if I delete all the failed marked snapshots and then run the backup again, and also if the stat check functionality has been reverted why I am receiving these errors, I have the latest version installed 3.3.2?
Click to expand...
As stated, the check will set the previous backup snapshot to a verify failed state, meaning that a new backup will be a full one. Since that re-uploads all the chunks, that might even "heal" some of the currently corrupt snapshots, if they reference such a chunk. So I suggest to simply run a verification of all the snapshots, but not delete any, then re-run all backup jobs which backed up to that datastore, and finally re-verify all again. That might help to recover some bad chunks.

Please note that the patch has been reverted, but it is not packaged just yet.
 
  • Like
Reactions: Johannes S
Yes, but without the skip verified... You should re-verify everything to be sure...

Edit: Just to be sure, this is related to your question regarding the re-verification, this is not the Verify new snapshots option. Latter has to be set on the datastores Options tab.
 
Last edited:
Chris said:
Yes, but without the skip verified... You should re-verify everything to be sure...

Edit: Just to be sure, this is related to your question regarding the re-verification, this is not the Verify new snapshots option. Latter has to be set on the datastores Options tab.
Click to expand...
I got a little confused here, can you please clarify, the verify new snapshots option in the datastore options tab is the one we will be using to verify all the snapshots we have or is there another option for it? From my understanding the verify new snapshots option will only verify the new snapshots that will come and not the previous snapshots we have.

Note: If we uncheck the option skip verified, will it check again for all the snapshots we have?
 
Hurara said:
I got a little confused here, can you please clarify, the verify new snapshots option in the datastore options tab is the one we will be using to verify all the snapshots we have or is there another option for it? From my understanding the verify new snapshots option will only verify the new snapshots that will come and not the previous snapshots we have.
Click to expand...
No, your understanding is correct. You will use a verify job to verify the consistency of all your snapshots. The mentioned option is for new snapshots only.

Hurara said:
Note: If we uncheck the option skip verified, will it check again for all the snapshots we have?
Click to expand...
Yes, that will re-verify all snapshots for the selected datastore.
 
  • Like
Reactions: Johannes S
Chris said:
No, your understanding is correct. You will use a verify job to verify the consistency of all your snapshots. The mentioned option is for new snapshots only.


Yes, that will re-verify all snapshots for the selected datastore.
Click to expand...
Thanks man, I will be doing it and getting back to you either way its solved or not. Thanks again!
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom