restore failed: Data blob has wrong CRC checksum.

rcarvalho

New Member
Aug 1, 2022
12
0
1
Hi there.

Recently my Promox VE Server (7.2 i beleive) broke down.
I've replaced with a new server and installed the most recent version of PVE.

I'm trying to restore VMs snapshot from the Backup Server but i'm getting "restore failed: Data blob has wrong CRC checksum."

In the task viewer of PBS i'm getting success message

2023-03-28T19:24:15+01:00: download chunk "/mnt/datastore/BACKUP-VM_FTP/.chunks/784a/784aebc32939d8260c14732ccee5f70edcfec288ff89a8e080dd083560d87aeb"
2023-03-28T19:24:16+01:00: reader finished successfully
2023-03-28T19:24:16+01:00: TASK OK

However on PVE i'm getting this:

Wiping dos signature on /dev/FTP/vm-100-disk-0.
Wiping atari signature on /dev/FTP/vm-100-disk-0.
Logical volume "vm-100-disk-0" created.
new volume ID is 'FTP:vm-100-disk-0'
WARNING: device /dev/sdc is an md component, not setting device for PV.
restore proxmox backup image: /usr/bin/pbs-restore --repository rcarvalho@pbs@192.168.88.196:BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /dev/FTP/vm-100-disk-0 --verbose --format raw
connecting to repository 'rcarvalho@pbs@192.168.88.196:BACKUP-VM_FTP'
open block backend for target '/dev/FTP/vm-100-disk-0'
starting to restore snapshot 'vm/100/2023-03-27T09:51:49Z'
download and verify backup index
progress 1% (read 9663676416 bytes, zeroes = 0% (54525952 bytes), duration 505 sec)
progress 2% (read 19327352832 bytes, zeroes = 0% (54525952 bytes), duration 1403 sec)
progress 3% (read 28991029248 bytes, zeroes = 0% (54525952 bytes), duration 2370 sec)
progress 4% (read 38654705664 bytes, zeroes = 0% (54525952 bytes), duration 3305 sec)
progress 5% (read 48318382080 bytes, zeroes = 0% (54525952 bytes), duration 4228 sec)
progress 6% (read 57982058496 bytes, zeroes = 0% (54525952 bytes), duration 5122 sec)
progress 7% (read 67645734912 bytes, zeroes = 0% (58720256 bytes), duration 6047 sec)
progress 8% (read 77309411328 bytes, zeroes = 0% (58720256 bytes), duration 6954 sec)
progress 9% (read 86973087744 bytes, zeroes = 0% (58720256 bytes), duration 7731 sec)
progress 10% (read 96636764160 bytes, zeroes = 0% (58720256 bytes), duration 8484 sec)
progress 11% (read 106300440576 bytes, zeroes = 0% (58720256 bytes), duration 9379 sec)
progress 12% (read 115964116992 bytes, zeroes = 0% (67108864 bytes), duration 10299 sec)
restore failed: Data blob has wrong CRC checksum.
WARNING: device /dev/sdc is an md component, not setting device for PV.
Logical volume "vm-100-disk-0" successfully removed
temporary volume 'FTP:vm-100-disk-0' sucessfuly removed
error before or during data restore, some or all disks were not completely restored. VM 100 state is NOT cleaned up.
TASK ERROR: command '/usr/bin/pbs-restore --repository rcarvalho@pbs@192.168.88.196:BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /dev/FTP/vm-100-disk-0 --verbose --format raw' failed: exit code 255

Aparently it's not lack of disk space as the snapshot has 960GB (aprox.) and the LVM has 1TB
Captura de ecrã 2023-03-28 212328.jpg
Thank you in advance
 
Did you run a recent verify job on your PBS to see if the backup snapshot got some chunks corrupted to rule out the your backup is damaged?
 
Hi @Dunuin
Thanks for the reply.

I had not done it, but i've done it today.

Aparently it had found corrupted chunk.

Task viewer: Datastore BACKUP-VM_FTP - Verification

OutputStatus

2023-03-29T09:28:18+01:00: verify datastore BACKUP-VM_FTP
2023-03-29T09:28:18+01:00: found 1 groups
2023-03-29T09:28:18+01:00: verify group BACKUP-VM_FTP:vm/100 (2 snapshots)
2023-03-29T09:28:18+01:00: verify BACKUP-VM_FTP:vm/100/2023-03-27T09:51:49Z
2023-03-29T09:28:18+01:00: check qemu-server.conf.blob
2023-03-29T09:28:18+01:00: check drive-virtio0.img.fidx
2023-03-29T11:17:54+01:00: can't verify chunk, load failed - store 'BACKUP-VM_FTP', unable to load chunk '784aebc32939d8260c14732ccee5f70edcfec288ff89a8e080dd083560d87aeb' - Data blob has wrong CRC checksum.
2023-03-29T11:17:54+01:00: corrupted chunk renamed to "/mnt/datastore/BACKUP-VM_FTP/.chunks/784a/784aebc32939d8260c14732ccee5f70edcfec288ff89a8e080dd083560d87aeb.0.bad"

I don't know how this happens, but is there anyway to recover?
 
Last edited:
if you don't have other backup snapshots that are not corrupt, you can use proxmox-backup-debug recover index /path/to/scsi0.fidx /path/to/datastore/.chunks --ignore-corrupt-chunks --skip-crc --ignore-missing-chunks --output-path some_file.raw to recover the uncorrupted parts, but in that case I would advices to use that to recover data and re-create the guest from scratch to avoid any undiscovered problems with the restored image (corrupt or missing chunks are replaced with zero bytes!)
 
  • Like
Reactions: Dunuin
Hi there.
I need some more help please with di proxmox-backup-debug command.

I've tryed to check the documentation but i'm not getting anyware.

I've tried to run this:
proxmox-backup-debug recover index /BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks --skip-crc --ignore-missing-chunks --output-path /BACKUP-VM_FTP/output.raw

but i'm getting this:

Error: parameter verification errors

parameter 'ignore-corrupt-chunks': missing parameter value.
parameter 'ignore-missing-chunks': missing parameter value.

Usage: proxmox-backup-debug recover index <file> <chunks> [OPTIONS]
<file> <string>
Path to the index file, either .fidx or .didx.

<chunks> <string>
Path to the directorty that contains the chunks, usually
<datastore>/.chunks.

Optional parameters:

--keyfile <string>
Path to a keyfile, if the data was encrypted, a keyfile is needed
for decryption.

--skip-crc <boolean> (default=false)
Skip the crc verification, increases the restore speed by lot.

What am i doing wrong?

Thanks
 
ah, sorry, you need to explicitly set them to 'true':

Code:
proxmox-backup-debug recover index /BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks true --skip-crc true --ignore-missing-chunks true --output-path /BACKUP-VM_FTP/output.raw
 
ok now i'm getting this:

root@PBS-SRV:~# proxmox-backup-debug recover index /BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks true --skip-crc true --ignore-missing-chunks true --output-path /BACKUP-VM_FTP/output.raw
Error: parameter verification errors

parameter 'ignore-corrupt-chunks': schema does not allow additional properties.
parameter 'ignore-missing-chunks': schema does not allow additional properties.
parameter 'output-path': schema does not allow additional properties.
 
ah, sorry, your command had a misplaced space - it should work with and without true if you fix that up:

/BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx

vs

/BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z/drive-virtio0.img.fidx
 
hmm not yet :)

root@PBS-SRV:~# proxmox-backup-debug recover index /BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z/drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks true --skip-crc true --ignore-missing-chunks true --output-path /BACKUP-VM_FTP/output.raw
Error: parameter verification errors

parameter 'ignore-corrupt-chunks': schema does not allow additional properties.
parameter 'ignore-missing-chunks': schema does not allow additional properties.
parameter 'output-path': schema does not allow additional properties.
 
yet another space (the first argument after index needs to be the full path to the .fidxfile)
 
root@PBS-SRV:~# proxmox-backup-debug recover index /BACKUP-VM_FTP/vm/100/2023-03-27T09:51:49Z/drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks true --skip-crc true --ignore-missing-chunks true --output-path /BACKUP-VM_FTP/output.raw
Error: parameter verification errors

parameter 'ignore-corrupt-chunks': schema does not allow additional properties.
parameter 'ignore-missing-chunks': schema does not allow additional properties.
parameter 'output-path': schema does not allow additional properties.
 
which PBS version do you have installed? IIRC the ignore parameters were added in 2.2.2-1
 
Hi again.

With PBS upgrade, it worked.

Kown i just need to discover how can i restore the raw output image.

Thanks
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!