restore failed: Data blob has wrong CRC checksum

[Mamiya_]

Hi all,
I'm starting my journey with virtualization and bump on problem that i don't really understand. Recently i set up PBS with data store at my Synology via NFS.
Backup process work fine but restoring not...
INFO: starting new backup job: vzdump 103 --mode snapshot --notes-template '{{guestname}}' --node pve --remove 0 --storage backup-pbs INFO: Starting Backup of VM 103 (qemu) INFO: Backup started at 2022-11-19 22:08:29 INFO: status = stopped INFO: backup mode: stop INFO: ionice priority: 7 INFO: VM Name: winstu01 INFO: include disk 'ide0' 'fast01:vm-103-disk-0' 30G INFO: include disk 'ide1' 'fast01:vm-103-disk-1' 50G INFO: creating Proxmox Backup Server archive 'vm/103/2022-11-19T21:08:29Z' INFO: starting kvm to execute backup task INFO: started backup task '9e7a86ae-88c1-4096-a4b5-30e5a82fe0a2' INFO: ide0: dirty-bitmap status: created new INFO: ide1: dirty-bitmap status: created new INFO: 0% (716.0 MiB of 80.0 GiB) in 3s, read: 238.7 MiB/s, write: 0 B/s ... INFO: 99% (79.8 GiB of 80.0 GiB) in 5m 23s, read: 676.0 MiB/s, write: 0 B/s INFO: 100% (80.0 GiB of 80.0 GiB) in 5m 24s, read: 232.0 MiB/s, write: 0 B/s INFO: backup is sparse: 5.80 GiB (7%) total zero data INFO: backup was done incrementally, reused 80.00 GiB (100%) INFO: transferred 80.00 GiB in 324 seconds (252.8 MiB/s) INFO: stopping kvm after backup task INFO: adding notes to backup INFO: Finished Backup of VM 103 (00:05:26) INFO: Backup finished at 2022-11-19 22:13:55 INFO: Backup job finished successfully TASK OK

when i do verification at backup i get
2022-11-19T22:51:29+01:00: verify group backup:vm/103 (2 snapshots) 2022-11-19T22:51:29+01:00: SKIPPED: verify backup:vm/103/2022-11-19T21:08:29Z (recently verified) 2022-11-19T22:51:29+01:00: percentage done: 50.00% (1/2 snapshots) 2022-11-19T22:51:29+01:00: verify backup:vm/103/2022-11-19T20:24:59Z 2022-11-19T22:51:29+01:00: check qemu-server.conf.blob 2022-11-19T22:51:29+01:00: check drive-ide1.img.fidx 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk '8facabdd922bd14e4b0d5b6ceed83ffe8272d18be2350ae790b6c99fb58c3ad8' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk 'c80b83d5f1369f12b9b66dbf662d3588f845c2c8885a802782f22bfeb802d3f8' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk '2058df4edc6d44943367a005b2a3cc75a2153f23537cb20dd7696a7d3544b752' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk '454335c449f61601d1000fc6d381cc6664ff4b2dbef39183cd1eed045cbc08d3' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk 'f47559ae2d5ff237efca4a5348e41e7e4e85b9c3bb85e2a0fe4cc91ad87dc03a' - No such file or directory (os error 2) 2022-11-19T22:57:21+01:00: verified 24754.07/48208.00 MiB in 352.15 seconds, speed 70.29/136.90 MiB/s (5 errors) 2022-11-19T22:57:21+01:00: verify backup:vm/103/2022-11-19T20:24:59Z/drive-ide1.img.fidx failed: chunks could not be verified 2022-11-19T22:57:21+01:00: check drive-ide0.img.fidx

same when i'm trying to restore backup

Logical volume "vm-102-disk-0" created. new volume ID is 'local-lvm:vm-102-disk-0' Logical volume "vm-102-disk-1" created. new volume ID is 'local-lvm:vm-102-disk-1' restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.88.107:backup vm/103/2022-11-19T20:24:59Z drive-ide0.img.fidx /dev/pve/vm-102-disk-0 --verbose --format raw --skip-zerof connecting to repository 'root@pam@192.168.88.107:backup' open block backend for target '/dev/pve/vm-102-disk-0' starting to restore snapshot 'vm/103/2022-11-19T20:24:59Z' download and verify backup index progress 1% (read 322961408 bytes, zeroes = 3% (12582912 bytes), duration 6 sec) ... progress 17% (read 5477761024 bytes, zeroes = 0% (16777216 bytes), duration 110 sec) restore failed: Data blob has wrong CRC checksum. Logical volume "vm-102-disk-0" successfully removed temporary volume 'local-lvm:vm-102-disk-0' sucessfuly removed Logical volume "vm-102-disk-1" successfully removed temporary volume 'local-lvm:vm-102-disk-1' sucessfuly removed error before or during data restore, some or all disks were not completely restored. VM 102 state is NOT cleaned up. TASK ERROR: command '/usr/bin/pbs-restore --repository root@pam@192.168.88.107:backup vm/103/2022-11-19T20:24:59Z drive-ide0.img.fidx /dev/pve/vm-102-disk-0 --verbose --format raw --skip-zero' failed: exit code 255

At my Synology i have at lest 2TB free space i'm attaching ss of my configuration. I'm running RAID 1 of 2 x 4TB Seagate NAS 5900 rpm.


mount at pbs machine
192.168.88.238:/volume1/backup /mnt/pbs-backup nfs4 rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.88.107,local_lock=none,addr=192.168.88.238 0 0 tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=807272k,nr_inodes=201818,mode=700,inode64 0 0
file permission
root@pbs:/mnt# ls -lah total 8.0K drwxr-xr-x 3 root root 4.0K Oct 23 21:53 . drwxr-xr-x 19 root root 4.0K Oct 13 17:33 .. drwxrwxr-x 1 backup backup 130 Nov 19 22:07 pbs-backup

 

[mokaz]

Hi there,

Just had the same symptoms on the PBS (itself) VM restore yesterday..

download and verify backup index
progress 1% (read 1375731712 bytes, zeroes = 99% (1367343104 bytes), duration 0 sec)
progress 2% (read 2751463424 bytes, zeroes = 99% (2743074816 bytes), duration 0 sec)
progress 3% (read 4127195136 bytes, zeroes = 99% (4118806528 bytes), duration 0 sec)
progress 4% (read 5498732544 bytes, zeroes = 99% (5490343936 bytes), duration 0 sec)
progress 5% (read 6874464256 bytes, zeroes = 99% (6866075648 bytes), duration 0 sec)
restore failed: Data blob has wrong CRC checksum.

Lucky me the "file" restore worked and I could export what I needed..
Other VMs restored okay, restored a few containers yesterday as well..

Have you setup the Synology according to this post?
https://forum.proxmox.com/threads/permssion-denied-os-error-13-on-nfs-share.74805/page-2#post-500057

Let me know,
Cheers,
m.

 

[Mamiya_]

Soooo what i did is:

1. changing all as guide was telling to do from your link, but last command won't work
proxmox-backup-client backup nodeABC-etc.pxar:/etc --repository 192.168.88.238:/volume1/pbs
so i have tried to create new data store via gui and after that swapping location and i stuck with error 13.

PBS have changed permissions from 777 to 100 for no reason i did several tries so is end with many test folders and accidentally start
rm -r ./* and you already know how it ended don't ask how i don't know either

Soooo i created new pbs machine, new network share [first ss with settings]
I have mounted it and guess what it worked!
Header Proxmox Virtual Environment 7.2-11 Datacenter Search: Logs Logical volume "vm-102-disk-0" created. new volume ID is 'local-lvm:vm-102-disk-0' Logical volume "vm-102-disk-1" created. new volume ID is 'local-lvm:vm-102-disk-1' restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.88.75:pbs vm/103/2022-11-20T15:29:36Z drive-ide0.img.fidx /dev/pve/vm-102-disk-0 --verbose --format raw --skip-zero connecting to repository 'root@pam@192.168.88.75:pbs' open block backend for target '/dev/pve/vm-102-disk-0' starting to restore snapshot 'vm/103/2022-11-20T15:29:36Z' download and verify backup index progress 1% (read 322961408 bytes, zeroes = 3% (12582912 bytes), duration 8 sec) ... progress 100% (read 32212254720 bytes, zeroes = 9% (3179282432 bytes), duration 838 sec) restore image complete (bytes=32212254720, duration=838.04s, speed=36.66MB/s) restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.88.75:pbs vm/103/2022-11-20T15:29:36Z drive-ide1.img.fidx /dev/pve/vm-102-disk-1 --verbose --format raw --skip-zero connecting to repository 'root@pam@192.168.88.75:pbs' open block backend for target '/dev/pve/vm-102-disk-1' starting to restore snapshot 'vm/103/2022-11-20T15:29:36Z' download and verify backup index progress 1% (read 536870912 bytes, zeroes = 3% (16777216 bytes), duration 20 sec) ... progress 100% (read 53687091200 bytes, zeroes = 5% (3053453312 bytes), duration 1375 sec) restore image complete (bytes=53687091200, duration=1375.95s, speed=37.21MB/s) rescan volumes... TASK OK

Windows machine with SQL works as i should! Now i'm waiting if also waiting for final verification in PBS

 

[Mamiya_]

got it!
2022-11-21T00:28:17+01:00: verify group pbs:vm/103 (1 snapshots) 2022-11-21T00:28:17+01:00: verify pbs:vm/103/2022-11-20T15:29:36Z 2022-11-21T00:28:17+01:00: check qemu-server.conf.blob 2022-11-21T00:28:17+01:00: check drive-ide1.img.fidx 2022-11-21T00:40:47+01:00: verified 24763.90/48228.00 MiB in 750.42 seconds, speed 33.00/64.27 MiB/s (0 errors) 2022-11-21T00:40:47+01:00: check drive-ide0.img.fidx 2022-11-21T00:48:05+01:00: verified 15548.82/27628.00 MiB in 436.95 seconds, speed 35.58/63.23 MiB/s (0 errors) 2022-11-21T00:48:05+01:00: percentage done: 100.00% (1/1 snapshots) 2022-11-21T00:48:05+01:00: TASK OK