[SOLVED] Restore failed: unable to parse raw blob - wrong magic

[Rotzbub]

PVE 7.4-16
PBS 2.4-2

What I did is a basic restore in PVE:​

• Go to Datacenter
• select the Backup-Storage 'disk'
• From BackupsI select one
• click on Restore
• Restore: VM I select the Storage (to where it will be restored) - click on Restore

Taskviewer - Logs 'task-pve-qmrestore...' (attached):

...
progress 19% (read 612032839680 bytes, zeroes = 6% (40319844352 bytes), duration 9827 sec)
progress 20% (read 644245094400 bytes, zeroes = 6% (42316333056 bytes), duration 10366 sec)
progress 21% (read 676457349120 bytes, zeroes = 6% (44304433152 bytes), duration 10904 sec)
restore failed: unable to parse raw blob - wrong magic
temporary volume 'tank:vm-100-disk-2' sucessfuly removed
temporary volume 'tank:vm-100-disk-0' sucessfuly removed
temporary volume 'tank:vm-100-disk-1' sucessfuly removed
error before or during data restore, some or all disks were not completely restored. VM 100 state is NOT cleaned up.
TASK ERROR: command '/usr/bin/pbs-restore --repository root@pam@127.0.0.1:diskname vm/111/2023-07-12T03:15:04Z drive-scsi1.img.fidx /dev/zvol/tank/vm-100-disk-2 --verbose --format raw --keyfile /etc/pve/priv/storage/diskname.enc --skip-zero' failed: exit code 255

What I tried also:​

• proxmox-backup-debug recover index drive-scsi1.img.fidx /mnt/datastore/diskname/.chunks --ignore-missing-chunks --keyfile /etc/pve/priv/storage/diskname.enc
  got same message:

  Error: unable to parse raw blob - wrong magic

• checked diskpace/inodes
• later backup (same vm) has the same error on the same percentage! - differnt backup (another vm) works
• file Restore is working

I only need one drive from the backup. Maybe there is another way to restore this to mount it to a vm ?

Thanks in advance!

 

[dcsapak]

can you try to do

proxmox-backup-debug inspect file <file>

for every file in the backup directory on pbs and post the outputs ?

 

[Rotzbub]

Thanks for the quick reply:

I shortened the chunks output...

# root@akroma:/mnt/datastore/diskname/vm/111/2023-07-14T03:15:06Z# ls -al
total 674782888
drwxr-xr-x  2 backup backup         4096 Jul 19 12:48 .
drwxr-xr-x 13 backup backup         4096 Jul 15 04:30 ..
-rw-r--r--  1 backup backup         2664 Jul 14 11:31 client.log.blob
-rw-r--r--  1 backup backup         4128 Jul 14 11:31 drive-efidisk0.img.fidx
-rw-r--r--  1 backup backup       135168 Jul 14 11:31 drive-scsi0.img.fidx
-rw-r--r--  1 root root 690952863744 Jul 18 11:31 drive-scsi1.img
-rw-r--r--  1 backup backup     24580096 Jul 14 11:31 drive-scsi1.img.fidx
-rw-r--r--  1 backup backup          621 Jul 14 11:31 index.json.blob
-rw-r--r--  1 backup backup          394 Jul 14 05:15 qemu-server.conf.blob

# proxmox-backup-debug inspect file client.log.blob
size: 2664
encryption: encrypt

# proxmox-backup-debug inspect file drive-efidisk0.img.fidx
size: 131072
creation time: Fri Jul 14 05:15:07 2023
chunks:
  "2795d9fadf103a4452b46a2870ee3dc6adca333c23e36319d208797a1a008629"
 
# proxmox-backup-debug inspect file drive-scsi0.img.fidx
size: 17179869184
creation time: Fri Jul 14 05:15:07 2023
chunks:
  "b8d7a99ec43da6f3562336c8ef7205d72d80de2dbddabeee82ffb3ed55870dd8"
  "7de95139467f45d34af5154548157c0f7d15cae0f144a69d5d7b71433468eb65"
  "554e3c765b41495c6fb3e832ce8e43d6248559348a95d2a416229bce316faf3b"
  ...
 
# proxmox-backup-debug inspect file drive-scsi1.img
Error: Only .blob, .fidx and .didx files may be inspected

# proxmox-backup-debug inspect file drive-scsi1.img.fidx | more
size: 3221225472000
creation time: Fri Jul 14 05:15:08 2023
chunks:
  "892b521b9d4094178df2d07a8a835ef84f7a847e362dafeee18e43b3e4bafa8a"
  "9e2588a91d847e592aa1861359a2db38df76862cdf5fce83dc819a7d4a249075"
  "11116e72fd74a72e95cbaeadde1e30b2d121ac1b925f356b43bdac645f4df494"
  "c351af23f4d04d7ea1ea636574d7acc215106c7752dec356982efb5e07721995"
  "1473b8e9c6699b5af8ff31cecc7fbdf43d17e8e8fdaea3e38e9a2d0d59fc34bc"
  "37a5a3c7e40ecd609f58462770b73cb6da4c06113de6d8bc44d95c99188558fa"
  "e390924675501e912723a75344401a3a6f77e9a770d55ab70d61f2e425cf3d2b"
  "9751bea447ba214723d32a8c33a6049a8c6a4a22093d36cbd8f961aa222b13ca"
  "6e13d2f0e8e279dea9d0afa857915344ccf57827f3d1a3f3597de902508fc92a"
  "650b39ceb103a8b3e86adb38513f77f5fab4a18c805bd985794b824a96ec999f"
  "f8a942a6819be2cf889ed3831fa0f27919691e5129903943d37bb35161c96e43"
  "57f85b79e7d2f6193be77df527eec8eb9ba45b5b762bf68cad094baa39058830"
  "4b584d9bcdb1d09c89b4d489780b6ae775c982638b02c97710d037fa7b4ca459"
  "0274be21c1f48b3fc1245f35903ebc2c802b22f5e314375238769b59f20aae85"
  "d0419e00db88a2a41d3c27a46b63efe27e64fd1ad23b9b8a70e583661b938089"
  "b51acdf1909ab94c99bd5be99d2c77bdc2d749458566fa04efd4bb86d77c1206"
  "dde15043dba5d9e4fcc07bb3c2ab3e43d7e72d03433265bf889187713324f30a"
  "409c11caa72d612becea542d2f36fdaa510b530168e41ec15c253d60d92eae99"
  ...

# proxmox-backup-debug inspect file index.json.blob
size: 621
encryption: none

# proxmox-backup-debug inspect file qemu-server.conf.blob
size: 394
encryption: encrypt

 

[dcsapak]

mhmmm.. why is there an .img file in there?

can you try to move that somewhere else? maybe it's trying to read that and fails

 

[Rotzbub]

true that,

here are some other ls outputs from that datastore - one of the other vm (112) i restored sucessfully
and i tried again with the one from 2023-07-12 - logs attached - same error

root@pve:/mnt/datastore/diskname/vm/111/2023-07-14T03:15:06Z# # ls -al ../../112/2023-07-14T09:31:25Z/
total 8284
drwxr-xr-x  2 backup backup    4096 Jul 14 13:04 .
drwxr-xr-x 10 backup backup    4096 Jul 15 04:30 ..
-rw-r--r--  1 backup backup    2673 Jul 14 13:04 client.log.blob
-rw-r--r--  1 backup backup  266240 Jul 14 13:04 drive-scsi0.img.fidx
-rw-r--r--  1 backup backup 8196096 Jul 14 13:04 drive-scsi1.img.fidx
-rw-r--r--  1 backup backup     570 Jul 14 13:04 index.json.blob
-rw-r--r--  1 backup backup     377 Jul 14 11:31 qemu-server.conf.blob
# ls -al ../2023-06-30T03:15:02Z/
total 24164
drwxr-xr-x  2 backup backup     4096 Jun 30 11:08 .
drwxr-xr-x 13 backup backup     4096 Jul 15 04:30 ..
-rw-r--r--  1 backup backup     2649 Jun 30 11:08 client.log.blob
-rw-r--r--  1 backup backup     4128 Jun 30 11:08 drive-efidisk0.img.fidx
-rw-r--r--  1 backup backup   135168 Jun 30 11:08 drive-scsi0.img.fidx
-rw-r--r--  1 backup backup 24580096 Jun 30 11:08 drive-scsi1.img.fidx
-rw-r--r--  1 backup backup      623 Jun 30 11:08 index.json.blob
-rw-r--r--  1 backup backup      394 Jun 30 05:15 qemu-server.conf.blob
# ls -al ../2022-12-30T04:15:01Z/
total 24164
drwxr-xr-x  2 backup backup     4096 Dec 30  2022 .
drwxr-xr-x 13 backup backup     4096 Jul 15 04:30 ..
-rw-r--r--  1 backup backup     2620 Dec 30  2022 client.log.blob
-rw-r--r--  1 backup backup     4128 Dec 30  2022 drive-efidisk0.img.fidx
-rw-r--r--  1 backup backup   135168 Dec 30  2022 drive-scsi0.img.fidx
-rw-r--r--  1 backup backup 24580096 Dec 30  2022 drive-scsi1.img.fidx
-rw-r--r--  1 backup backup      622 Dec 30  2022 index.json.blob
-rw-r--r--  1 backup backup      390 Dec 30  2022 qemu-server.conf.blob
Z# ls -al ../2023-07-12T03:15:04Z/
total 24164
drwxr-xr-x  2 backup backup     4096 Jul 12 11:04 .
drwxr-xr-x 13 backup backup     4096 Jul 15 04:30 ..
-rw-r--r--  1 backup backup     2695 Jul 12 11:04 client.log.blob
-rw-r--r--  1 backup backup     4128 Jul 12 11:04 drive-efidisk0.img.fidx
-rw-r--r--  1 backup backup   135168 Jul 12 11:04 drive-scsi0.img.fidx
-rw-r--r--  1 backup backup 24580096 Jul 12 11:04 drive-scsi1.img.fidx
-rw-r--r--  1 backup backup      618 Jul 12 11:04 index.json.blob
-rw-r--r--  1 backup backup      394 Jul 12 05:15 qemu-server.conf.blob

 

[dcsapak]

is there anything in the pbs journal/syslog during a restore that might indicate the problem?

 

[dcsapak]

ok a colleague reminded me that this also can happen if the encrypted chunks are corrupt

you can check how many chunks are impacted with the proxmox-backup-debug recover command but with the '--ignore-corrupt-chunks' parameter (this replaces corrupt chunks with zeroes)

did you run a verify recently? this should have caught that and marked the chunk as corrupted

 

[Rotzbub]

Yess,Thanks ! The Error is due to a corrupt chunk it seems:

root@pve:/mnt/datastore/diskname/vm/111/2023-07-12T03:15:04Z# proxmox-backup-debug recover index drive-scsi1.img.fidx /mnt/datastore/diskname/.chunks --ignore-missing-chunks --keyfile /etc/pve/priv/storage/disk8tb1.enc --ignore-corrupt-chunks --output-path /tank/drive-scsi1.img
WARN: chunk "/mnt/datastore/diskname/.chunks/0257/0257bcf3021ea63daf7a272ba5b8e698ca188b65f9680d780b0fa06e0f7259f2" is corrupt - unable to parse raw blob - wrong magic
WARN: replacing output file 690952863744..690957058048 with '\0'

I did a proxmox-backup-debug recover index but only with --ignore-missing-chunks not --ignore-corrupt-chunks. Also thats why there was an .img file in the folder.

 

[Rotzbub]

• I will now import the created .img - qm importdisk 111 drive-scsi1.img tank --format raw
• And run a (in PBS) a verification of the disk to solve this in the future
• As I have another PBS backup of this vm, maybe I will restore from there or can crosscheck to fix this corrupted chunk - but that would be another thread (will link if I found something)