restore failed: Data blob has wrong CRC checksum

Mamiya_

New Member
Oct 14, 2022
4
4
3
Hi all,
I'm starting my journey with virtualization and bump on problem that i don't really understand. Recently i set up PBS with data store at my Synology via NFS.
Backup process work fine but restoring not...
INFO: starting new backup job: vzdump 103 --mode snapshot --notes-template '{{guestname}}' --node pve --remove 0 --storage backup-pbs INFO: Starting Backup of VM 103 (qemu) INFO: Backup started at 2022-11-19 22:08:29 INFO: status = stopped INFO: backup mode: stop INFO: ionice priority: 7 INFO: VM Name: winstu01 INFO: include disk 'ide0' 'fast01:vm-103-disk-0' 30G INFO: include disk 'ide1' 'fast01:vm-103-disk-1' 50G INFO: creating Proxmox Backup Server archive 'vm/103/2022-11-19T21:08:29Z' INFO: starting kvm to execute backup task INFO: started backup task '9e7a86ae-88c1-4096-a4b5-30e5a82fe0a2' INFO: ide0: dirty-bitmap status: created new INFO: ide1: dirty-bitmap status: created new INFO: 0% (716.0 MiB of 80.0 GiB) in 3s, read: 238.7 MiB/s, write: 0 B/s ... INFO: 99% (79.8 GiB of 80.0 GiB) in 5m 23s, read: 676.0 MiB/s, write: 0 B/s INFO: 100% (80.0 GiB of 80.0 GiB) in 5m 24s, read: 232.0 MiB/s, write: 0 B/s INFO: backup is sparse: 5.80 GiB (7%) total zero data INFO: backup was done incrementally, reused 80.00 GiB (100%) INFO: transferred 80.00 GiB in 324 seconds (252.8 MiB/s) INFO: stopping kvm after backup task INFO: adding notes to backup INFO: Finished Backup of VM 103 (00:05:26) INFO: Backup finished at 2022-11-19 22:13:55 INFO: Backup job finished successfully TASK OK

when i do verification at backup i get
2022-11-19T22:51:29+01:00: verify group backup:vm/103 (2 snapshots) 2022-11-19T22:51:29+01:00: SKIPPED: verify backup:vm/103/2022-11-19T21:08:29Z (recently verified) 2022-11-19T22:51:29+01:00: percentage done: 50.00% (1/2 snapshots) 2022-11-19T22:51:29+01:00: verify backup:vm/103/2022-11-19T20:24:59Z 2022-11-19T22:51:29+01:00: check qemu-server.conf.blob 2022-11-19T22:51:29+01:00: check drive-ide1.img.fidx 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk '8facabdd922bd14e4b0d5b6ceed83ffe8272d18be2350ae790b6c99fb58c3ad8' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk 'c80b83d5f1369f12b9b66dbf662d3588f845c2c8885a802782f22bfeb802d3f8' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk '2058df4edc6d44943367a005b2a3cc75a2153f23537cb20dd7696a7d3544b752' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk '454335c449f61601d1000fc6d381cc6664ff4b2dbef39183cd1eed045cbc08d3' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk 'f47559ae2d5ff237efca4a5348e41e7e4e85b9c3bb85e2a0fe4cc91ad87dc03a' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: verified 24754.07/48208.00 MiB in 352.15 seconds, speed 70.29/136.90 MiB/s (5 errors) 2022-11-19T22:57:21+01:00: verify backup:vm/103/2022-11-19T20:24:59Z/drive-ide1.img.fidx failed: chunks could not be verified 2022-11-19T22:57:21+01:00: check drive-ide0.img.fidx

same when i'm trying to restore backup

Logical volume "vm-102-disk-0" created. new volume ID is 'local-lvm:vm-102-disk-0' Logical volume "vm-102-disk-1" created. new volume ID is 'local-lvm:vm-102-disk-1' restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.88.107:backup vm/103/2022-11-19T20:24:59Z drive-ide0.img.fidx /dev/pve/vm-102-disk-0 --verbose --format raw --skip-zerof connecting to repository 'root@pam@192.168.88.107:backup' open block backend for target '/dev/pve/vm-102-disk-0' starting to restore snapshot 'vm/103/2022-11-19T20:24:59Z' download and verify backup index progress 1% (read 322961408 bytes, zeroes = 3% (12582912 bytes), duration 6 sec) ... progress 17% (read 5477761024 bytes, zeroes = 0% (16777216 bytes), duration 110 sec) restore failed: Data blob has wrong CRC checksum. Logical volume "vm-102-disk-0" successfully removed temporary volume 'local-lvm:vm-102-disk-0' sucessfuly removed Logical volume "vm-102-disk-1" successfully removed temporary volume 'local-lvm:vm-102-disk-1' sucessfuly removed error before or during data restore, some or all disks were not completely restored. VM 102 state is NOT cleaned up. TASK ERROR: command '/usr/bin/pbs-restore --repository root@pam@192.168.88.107:backup vm/103/2022-11-19T20:24:59Z drive-ide0.img.fidx /dev/pve/vm-102-disk-0 --verbose --format raw --skip-zero' failed: exit code 255

At my Synology i have at lest 2TB free space i'm attaching ss of my configuration. I'm running RAID 1 of 2 x 4TB Seagate NAS 5900 rpm.


mount at pbs machine
192.168.88.238:/volume1/backup /mnt/pbs-backup nfs4 rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.88.107,local_lock=none,addr=192.168.88.238 0 0 tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=807272k,nr_inodes=201818,mode=700,inode64 0 0
file permission
root@pbs:/mnt# ls -lah total 8.0K drwxr-xr-x 3 root root 4.0K Oct 23 21:53 . drwxr-xr-x 19 root root 4.0K Oct 13 17:33 .. drwxrwxr-x 1 backup backup 130 Nov 19 22:07 pbs-backup
 

Attachments

  • 2022-11-19_23h02_11.png
    2022-11-19_23h02_11.png
    21.9 KB · Views: 1
  • 2022-11-19_23h03_09.png
    2022-11-19_23h03_09.png
    14 KB · Views: 1
Last edited:
Hi there,

Just had the same symptoms on the PBS (itself) VM restore yesterday..

Code:
download and verify backup index
progress 1% (read 1375731712 bytes, zeroes = 99% (1367343104 bytes), duration 0 sec)
progress 2% (read 2751463424 bytes, zeroes = 99% (2743074816 bytes), duration 0 sec)
progress 3% (read 4127195136 bytes, zeroes = 99% (4118806528 bytes), duration 0 sec)
progress 4% (read 5498732544 bytes, zeroes = 99% (5490343936 bytes), duration 0 sec)
progress 5% (read 6874464256 bytes, zeroes = 99% (6866075648 bytes), duration 0 sec)
restore failed: Data blob has wrong CRC checksum.

Lucky me the "file" restore worked and I could export what I needed..
Other VMs restored okay, restored a few containers yesterday as well..

Have you setup the Synology according to this post?
https://forum.proxmox.com/threads/permssion-denied-os-error-13-on-nfs-share.74805/page-2#post-500057

Let me know,
Cheers,
m.
 
Soooo what i did is:

1. changing all as guide was telling to do from your link, but last command won't work
proxmox-backup-client backup nodeABC-etc.pxar:/etc --repository 192.168.88.238:/volume1/pbs
so i have tried to create new data store via gui and after that swapping location and i stuck with error 13.

PBS have changed permissions from 777 to 100 for no reason i did several tries so is end with many test folders and accidentally start
rm -r ./* and you already know how it ended :) don't ask how i don't know either

Soooo i created new pbs machine, new network share [first ss with settings]
I have mounted it and guess what it worked!
Header Proxmox Virtual Environment 7.2-11 Datacenter Search: Logs Logical volume "vm-102-disk-0" created. new volume ID is 'local-lvm:vm-102-disk-0' Logical volume "vm-102-disk-1" created. new volume ID is 'local-lvm:vm-102-disk-1' restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.88.75:pbs vm/103/2022-11-20T15:29:36Z drive-ide0.img.fidx /dev/pve/vm-102-disk-0 --verbose --format raw --skip-zero connecting to repository 'root@pam@192.168.88.75:pbs' open block backend for target '/dev/pve/vm-102-disk-0' starting to restore snapshot 'vm/103/2022-11-20T15:29:36Z' download and verify backup index progress 1% (read 322961408 bytes, zeroes = 3% (12582912 bytes), duration 8 sec) ... progress 100% (read 32212254720 bytes, zeroes = 9% (3179282432 bytes), duration 838 sec) restore image complete (bytes=32212254720, duration=838.04s, speed=36.66MB/s) restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.88.75:pbs vm/103/2022-11-20T15:29:36Z drive-ide1.img.fidx /dev/pve/vm-102-disk-1 --verbose --format raw --skip-zero connecting to repository 'root@pam@192.168.88.75:pbs' open block backend for target '/dev/pve/vm-102-disk-1' starting to restore snapshot 'vm/103/2022-11-20T15:29:36Z' download and verify backup index progress 1% (read 536870912 bytes, zeroes = 3% (16777216 bytes), duration 20 sec) ... progress 100% (read 53687091200 bytes, zeroes = 5% (3053453312 bytes), duration 1375 sec) restore image complete (bytes=53687091200, duration=1375.95s, speed=37.21MB/s) rescan volumes... TASK OK

Windows machine with SQL works as i should! Now i'm waiting if also waiting for final verification in PBS
 

Attachments

  • 2022-11-21_00h24_33.png
    2022-11-21_00h24_33.png
    44.5 KB · Views: 1
got it!
2022-11-21T00:28:17+01:00: verify group pbs:vm/103 (1 snapshots) 2022-11-21T00:28:17+01:00: verify pbs:vm/103/2022-11-20T15:29:36Z 2022-11-21T00:28:17+01:00: check qemu-server.conf.blob 2022-11-21T00:28:17+01:00: check drive-ide1.img.fidx 2022-11-21T00:40:47+01:00: verified 24763.90/48228.00 MiB in 750.42 seconds, speed 33.00/64.27 MiB/s (0 errors) 2022-11-21T00:40:47+01:00: check drive-ide0.img.fidx 2022-11-21T00:48:05+01:00: verified 15548.82/27628.00 MiB in 436.95 seconds, speed 35.58/63.23 MiB/s (0 errors) 2022-11-21T00:48:05+01:00: percentage done: 100.00% (1/1 snapshots) 2022-11-21T00:48:05+01:00: TASK OK
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!