restore failed: Data blob has wrong CRC checksum.

rcarvalho · Mar 28, 2023

Hi there.

Recently my Promox VE Server (7.2 i beleive) broke down.
I've replaced with a new server and installed the most recent version of PVE.

I'm trying to restore VMs snapshot from the Backup Server but i'm getting "restore failed: Data blob has wrong CRC checksum."

In the task viewer of PBS i'm getting success message

2023-03-28T19:24:15+01:00: download chunk "/mnt/datastore/BACKUP-VM_FTP/.chunks/784a/784aebc32939d8260c14732ccee5f70edcfec288ff89a8e080dd083560d87aeb"
2023-03-28T19:24:16+01:00: reader finished successfully
2023-03-28T19:24:16+01:00: TASK OK

However on PVE i'm getting this:

Wiping dos signature on /dev/FTP/vm-100-disk-0.
Wiping atari signature on /dev/FTP/vm-100-disk-0.
Logical volume "vm-100-disk-0" created.
new volume ID is 'FTP:vm-100-disk-0'
WARNING: device /dev/sdc is an md component, not setting device for PV.
restore proxmox backup image: /usr/bin/pbs-restore --repository rcarvalho@pbs@192.168.88.196:BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /dev/FTP/vm-100-disk-0 --verbose --format raw
connecting to repository 'rcarvalho@pbs@192.168.88.196:BACKUP-VM_FTP'
open block backend for target '/dev/FTP/vm-100-disk-0'
starting to restore snapshot 'vm/100/2023-03-27T09:51:49Z'
download and verify backup index
progress 1% (read 9663676416 bytes, zeroes = 0% (54525952 bytes), duration 505 sec)
progress 2% (read 19327352832 bytes, zeroes = 0% (54525952 bytes), duration 1403 sec)
progress 3% (read 28991029248 bytes, zeroes = 0% (54525952 bytes), duration 2370 sec)
progress 4% (read 38654705664 bytes, zeroes = 0% (54525952 bytes), duration 3305 sec)
progress 5% (read 48318382080 bytes, zeroes = 0% (54525952 bytes), duration 4228 sec)
progress 6% (read 57982058496 bytes, zeroes = 0% (54525952 bytes), duration 5122 sec)
progress 7% (read 67645734912 bytes, zeroes = 0% (58720256 bytes), duration 6047 sec)
progress 8% (read 77309411328 bytes, zeroes = 0% (58720256 bytes), duration 6954 sec)
progress 9% (read 86973087744 bytes, zeroes = 0% (58720256 bytes), duration 7731 sec)
progress 10% (read 96636764160 bytes, zeroes = 0% (58720256 bytes), duration 8484 sec)
progress 11% (read 106300440576 bytes, zeroes = 0% (58720256 bytes), duration 9379 sec)
progress 12% (read 115964116992 bytes, zeroes = 0% (67108864 bytes), duration 10299 sec)
restore failed: Data blob has wrong CRC checksum.
WARNING: device /dev/sdc is an md component, not setting device for PV.
Logical volume "vm-100-disk-0" successfully removed
temporary volume 'FTP:vm-100-disk-0' sucessfuly removed
error before or during data restore, some or all disks were not completely restored. VM 100 state is NOT cleaned up.
TASK ERROR: command '/usr/bin/pbs-restore --repository rcarvalho@pbs@192.168.88.196:BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /dev/FTP/vm-100-disk-0 --verbose --format raw' failed: exit code 255

Aparently it's not lack of disk space as the snapshot has 960GB (aprox.) and the LVM has 1TB

Thank you in advance

Dunuin · Mar 28, 2023

Did you run a recent verify job on your PBS to see if the backup snapshot got some chunks corrupted to rule out the your backup is damaged?

rcarvalho · Mar 29, 2023

Hi @Dunuin
Thanks for the reply.

I had not done it, but i've done it today.

Aparently it had found corrupted chunk.

Task viewer: Datastore BACKUP-VM_FTP - Verification

OutputStatus

2023-03-29T09:28:18+01:00: verify datastore BACKUP-VM_FTP
2023-03-29T09:28:18+01:00: found 1 groups
2023-03-29T09:28:18+01:00: verify group BACKUP-VM_FTP:vm/100 (2 snapshots)
2023-03-29T09:28:18+01:00: verify BACKUP-VM_FTP:vm/100/2023-03-27T09:51:49Z
2023-03-29T09:28:18+01:00: check qemu-server.conf.blob
2023-03-29T09:28:18+01:00: check drive-virtio0.img.fidx
2023-03-29T11:17:54+01:00: can't verify chunk, load failed - store 'BACKUP-VM_FTP', unable to load chunk '784aebc32939d8260c14732ccee5f70edcfec288ff89a8e080dd083560d87aeb' - Data blob has wrong CRC checksum.
2023-03-29T11:17:54+01:00: corrupted chunk renamed to "/mnt/datastore/BACKUP-VM_FTP/.chunks/784a/784aebc32939d8260c14732ccee5f70edcfec288ff89a8e080dd083560d87aeb.0.bad"

I don't know how this happens, but is there anyway to recover?

fabian · Mar 29, 2023

if you don't have other backup snapshots that are not corrupt, you can use

proxmox-backup-debug recover index /path/to/scsi0.fidx /path/to/datastore/.chunks --ignore-corrupt-chunks --skip-crc --ignore-missing-chunks --output-path some_file.raw

to recover the uncorrupted parts, but in that case I would advices to use that to recover data and re-create the guest from scratch to avoid any undiscovered problems with the restored image (corrupt or missing chunks are replaced with zero bytes!)

rcarvalho · Mar 30, 2023

Hi there.
I need some more help please with di proxmox-backup-debug command.

I've tryed to check the documentation but i'm not getting anyware.

I've tried to run this:
proxmox-backup-debug recover index /BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks --skip-crc --ignore-missing-chunks --output-path /BACKUP-VM_FTP/output.raw

but i'm getting this:

Error: parameter verification errors

parameter 'ignore-corrupt-chunks': missing parameter value.
parameter 'ignore-missing-chunks': missing parameter value.

Usage: proxmox-backup-debug recover index <file> <chunks> [OPTIONS]
<file> <string>
Path to the index file, either .fidx or .didx.

<chunks> <string>
Path to the directorty that contains the chunks, usually
<datastore>/.chunks.

Optional parameters:

--keyfile <string>
Path to a keyfile, if the data was encrypted, a keyfile is needed
for decryption.

--skip-crc <boolean> (default=false)
Skip the crc verification, increases the restore speed by lot.

What am i doing wrong?

Thanks

fabian · Mar 30, 2023

ah, sorry, you need to explicitly set them to 'true':

Code:

proxmox-backup-debug recover index /BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks true --skip-crc true --ignore-missing-chunks true --output-path /BACKUP-VM_FTP/output.raw

rcarvalho · Mar 30, 2023

ok now i'm getting this:

root@PBS-SRV:~# proxmox-backup-debug recover index /BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks true --skip-crc true --ignore-missing-chunks true --output-path /BACKUP-VM_FTP/output.raw
Error: parameter verification errors

parameter 'ignore-corrupt-chunks': schema does not allow additional properties.
parameter 'ignore-missing-chunks': schema does not allow additional properties.
parameter 'output-path': schema does not allow additional properties.

fabian · Mar 30, 2023

ah, sorry, your command had a misplaced space - it should work with and without true if you fix that up:

/BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z drive-virtio0.img.fidx

vs

/BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z/drive-virtio0.img.fidx

rcarvalho · Mar 30, 2023

hmm not yet

root@PBS-SRV:~# proxmox-backup-debug recover index /BACKUP-VM_FTP vm/100/2023-03-27T09:51:49Z/drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks true --skip-crc true --ignore-missing-chunks true --output-path /BACKUP-VM_FTP/output.raw
Error: parameter verification errors

parameter 'ignore-corrupt-chunks': schema does not allow additional properties.
parameter 'ignore-missing-chunks': schema does not allow additional properties.
parameter 'output-path': schema does not allow additional properties.

fabian · Mar 30, 2023

yet another space (the first argument after index needs to be the full path to the .fidxfile)

rcarvalho · Mar 30, 2023

root@PBS-SRV:~# proxmox-backup-debug recover index /BACKUP-VM_FTP/vm/100/2023-03-27T09:51:49Z/drive-virtio0.img.fidx /mnt/datastore/BACKUP-VM_FTP/.chunks --ignore-corrupt-chunks true --skip-crc true --ignore-missing-chunks true --output-path /BACKUP-VM_FTP/output.raw
Error: parameter verification errors

parameter 'ignore-corrupt-chunks': schema does not allow additional properties.
parameter 'ignore-missing-chunks': schema does not allow additional properties.
parameter 'output-path': schema does not allow additional properties.

fabian · Mar 30, 2023

which PBS version do you have installed? IIRC the ignore parameters were added in 2.2.2-1

rcarvalho · Mar 30, 2023

Backup Server 2.1-1

It's better to upgrade?

Dunuin · Mar 30, 2023

rcarvalho said:
It's better to upgrade?

You should always keep your OSs updated if you don't want to run unsecure systems. If you are running 2.1 you got at least 1 year of unpatched vulnerabilities.

rcarvalho · Mar 30, 2023

Going for it and i'll give feedback later.
Thank you

rcarvalho · Mar 31, 2023

Hi again.

With PBS upgrade, it worked.

Kown i just need to discover how can i restore the raw output image.

Thanks

fabian · Apr 3, 2023

you can import it into a VM to (attempt to) recover files/data: https://pve.proxmox.com/pve-docs/chapter-qm.html#_adding_an_external_disk_image_to_a_virtual_machine

Search

Search

restore failed: Data blob has wrong CRC checksum.

rcarvalho

Member

Dunuin

Distinguished Member

rcarvalho

Member

fabian

Proxmox Staff Member

rcarvalho

Member

fabian

Proxmox Staff Member

rcarvalho

Member

fabian

Proxmox Staff Member

rcarvalho

Member

fabian

Proxmox Staff Member

rcarvalho

Member

fabian

Proxmox Staff Member

rcarvalho

Member

Dunuin

Distinguished Member

rcarvalho

Member

rcarvalho

Member

fabian

Proxmox Staff Member