Cannot restore VM from backup

[khangnch]

Hi everyone.

I have a VM that cannot be restored from its backup. I restore the VM then it shows error as below:

TASK ERROR: error during cfs-locked 'storage-ssd' operation: rbd create vm-1030-cloudinit' error: rbd: create error: 2022-05-30 16:44:06.941120 7f12949760c0 -1 librbd: rbd image vm-1030-cloudinit already exists

I have already delete this image from command line but it got into a new error:

TASK ERROR: command '/usr/bin/pbs-restore --repository server1@pbs@172.16.11.240:server1 vm/1030/2022-05-28T15:42:36Z drive-virtio0.img.fidx 'rbd:server1/vm-1030-disk-0:mon_host=10.10.10.111;10.10.10.112;10.10.10.113:auth_supported=cephx:id=admin:keyring=/etc/pve/priv/ceph/ssd.keyring' --verbose --format raw --skip-zero' failed: exit code 255

In backup server, I saw the log belows:

can't verify chunk, stat failed - No such file or directory (os error 2)

Please suggest me which can cause this error and how to fix it.

 

[khangnch]

For more information, the error log appear when I restore as below:

restore failed: reading file "/backup/proxmox/server1/.chunks/b8cd/b8cd5efb4254adf2e0d71341834358d3fb6b2a5547d8131ea641cb503cefcb52" failed: No such file or directory (os error 2)
temporary volume 'backup:144/vm-144-disk-0.raw' sucessfuly removed
TASK ERROR: command '/usr/bin/pbs-restore --repository server1@pbs@172.16.11.240:b101 vm/1030/2022-05-28T15:42:36Z drive-virtio0.img.fidx /mnt/pve/backup/images/144/vm-144-disk-0.raw --verbose --format raw --skip-zero' failed: exit code 25

Please advice me if there are any tips to fix the problem. I need to restore the data from the backup.

Thank you very much for the help.

 

[fabian]

you are missing a chunk (possible it got moved out of the way by a verify task because it got corrupted? or someone manually deleted it?), so this snapshot is not restorable with regular means.

the next release of proxmox-backup-server will have a new feature for the proxmox-backup-debug binary that allows partially recovering such files:

https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=53435bc4d51539e6a23d6602d7965a563aba4d09

but you should find out what happened (check past verification logs for that group/datastore!)

 

[khangnch]

Hi,

Thank you very much for your reply.

The Verify New Snapshots was No before. I think this can cause this problem.



When I tried to verify the backup, there came up error:

 

[fabian]

you don't (necessarily) need to verify each snapshot directly after creation - just somewhat regularly to catch anything being amiss.

 

[khangnch]

you don't (necessarily) need to verify each snapshot directly after creation - just somewhat regularly to catch anything being amiss.

Hi,

I still cannot specify why the chunk file was corrupt.

But when I use the proxmox-backup-debug binary, it requires the chunk file location.

proxmox-backup-debug recover index <file> <chunks>

I am new to PBS, so please specify more, I will be very appreciative.

When I try to restore from the backup again. It comes up the error:

restore failed: reading file "/backup/proxmox/server1/.chunks/b8cd/b8cd5efb4254adf2e0d71341834358d3fb6b2a5547d8131ea641cb503cefcb52" failed: No such file or directory (os error 2)

Thank you very much.

 

[fabian]

the code I linked is not yet released, so you have to wait until it is available in the repositories (or build yourself).

the command will (then) be:

proxmox-backup-debug recover index /path/to/file.fidx /path/to/.chunks --ignore-missing-chunks --output-path /path/to/output.file

 

[khangnch]

the code I linked is not yet released, so you have to wait until it is available in the repositories (or build yourself).

the command will (then) be:

proxmox-backup-debug recover index /path/to/file.fidx /path/to/.chunks --ignore-missing-chunks --output-path /path/to/output.file

Thank you for your help.

Can you let me know when the new release will be available?

I will prepare to test this.

 

[fabian]

version 2.2.2-1 will contain the feature, I can't tell you how fast it will be rolled out.

 

[AnNH]

Hi,

After I use commad:

proxmox-backup-debug recover index /backup/proxmox/server1/vm/1030/2022-05-28T15:42:36Z/drive-virtio0.img.fidx /backup/proxmox/server1/.chunks --ignore-missing-chunks

Then I tried restore VM in Proxmox VE I got this error, can you help ?

Logical volume "vm-1030-cloudinit" created.
new volume ID is 'local-lvm:vm-1030-cloudinit'
  Logical volume "vm-1030-disk-0" created.
new volume ID is 'local-lvm:vm-1030-disk-0'
restore proxmox backup image: /usr/bin/pbs-restore --repository server1@pbs@172.16.11.240:server1vm/1030/2022-05-28T15:42:36Z drive-virtio0.img.fidx /dev/pve/vm-1030-disk-0 --verbose --format raw --skip-zero
connecting to repository 'sevrer1@pbs@172.16.11.240:sevrer1'
open block backend for target '/dev/pve/vm-1030-disk-0'
starting to restore snapshot 'vm/1030/2022-05-28T15:42:36Z'
download and verify backup index
restore failed: unable to parse raw blob - wrong magic
  Logical volume "vm-1030-disk-0" successfully removed
temporary volume 'local-lvm:vm-1030-disk-0' sucessfuly removed
TASK ERROR: command '/usr/bin/pbs-restore --repository server1@pbs@172.16.11.240:server1vm/1030/2022-05-28T15:42:36Z drive-virtio0.img.fidx /dev/pve/vm-1030-disk-0 --verbose --format raw --skip-zero' failed: exit code 255

Thanks and Regards !

 

[fabian]

what does the PBS task log corresponding to the restore say?

 

[AnNH]

Hi,

When running the command on pbs, the system detects the missing chunk file and replaces it with a value of 0. Then it was over that we did the restore. Do I need to do anything else before restoring?

what does the PBS task log corresponding to the restore say?

Can I do that command again ?

And after trying to restore to another VMID on an NFS directory associated with PVE, I got this error:

starting to restore snapshot 'vm/1030/2022-05-28T15:42:36Z'

download and verify backup index

restore failed: protocol error: stream no longer needed

temporary volume 'backup:10004/vm-10004-disk-0.raw' sucessfuly removed

TASK ERROR: command '/usr/bin/pbs-restore --repository b101@pbs@172.16.11.240:b101 vm/1030/2022-05-28T15:42:36Z drive-virtio0.img.fidx /mnt/pve/backup/images/10004/vm-10004-disk-0.raw --verbose --format raw --skip-zero' failed: exit code 255

Thanks and Regards !

 

[fabian]

ah - you misunderstood what the recover command does - it doesn't replace the chunk in the chunk store (that wouldn't help at all - PBS would still rightfully treat it as corrupt), it generates an output file (by default at the same path as the index, but without the .fidx extension) that contains the recovered data (which you can then transfer to PVE and import into a VM - via qm importdisk .. or qm create .. TARGETSTORAGE:0,import-source=/path/to/recovered/image,..)

 

[khangnch]

Hi.

We use the command to create the output file and import to a new VM but there was no data on the disk. The size of new disk and the old one are the same.

We use the command as below to import the new disk:

qm importdisk 14477 drivee-virtio0.img ssd -format raw

 

[fabian]

just to make sure - you then also need to attach the disk to the VM in order to "see" it inside.. how many chunks were missing/corrupt? there is a chance there's too much gone for recovery to work..

 

[khangnch]

Hi.

We have attached the disk and check the data integrity. We cannot specify how many chunks were missing but when we execute the command, there seem a lot of them.

WARN: chunk "/backup/proxmox/server1/.chunks/b397/b39794568f61e9e7b0d838a6483f2df6790f466e8399a918ecc3eee85d2736ac" is missing
WARN: replacing output file 6819938304..6824132608 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/8137/8137ce22e56441152baafa4c41bc15b85c11c07a1fc494f414f2c529553ebc31" is missing
WARN: replacing output file 6824132608..6828326912 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/0c63/0c633e2a38d2e23d13765a3d83e2dd8c78d12518bd0f300b7fe2cd9206d6ccd9" is missing
WARN: replacing output file 6828326912..6832521216 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/8649/86494900c790b9b90093a6acd788e14742f90a2b8314462f5db8361ac053b6d3" is missing
WARN: replacing output file 6832521216..6836715520 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/cc15/cc15e3de010de684be464c0ba05bbcdd5755e3d622ac4caf370fb21a7304a6e9" is missing
WARN: replacing output file 6836715520..6840909824 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/e1a3/e1a3dff16d7064b2a36659b413f2d28bd0b8bd90406dec3c47c4e1947809c594" is missing
WARN: replacing output file 6840909824..6845104128 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/aa42/aa42089ec5474d4ede90c89f2ce9c306c1bcd064ba48b35bdb02f21c49c726fb" is missing
WARN: replacing output file 6845104128..6849298432 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/a28b/a28bbc71e74c6a09e50e63c7a5e64563c495688895a6fb3af679db933fdfecf6" is missing
WARN: replacing output file 6849298432..6853492736 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/cd84/cd848b899b1455765dbe244ebdf0b7af8f30c87889bde8b206adf58810af6813" is missing
WARN: replacing output file 6853492736..6857687040 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/abd2/abd27a1b5f36b7fb823df77c299e41a4416a1531786c6fad4cb318e671e53dc5" is missing
WARN: replacing output file 6857687040..6861881344 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/5a37/5a3766267f9ead9f8505540820f5666b077254073414d7d37cb66dd7f8eeebf2" is missing
WARN: replacing output file 6861881344..6866075648 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/a003/a0031e9cdab32b46a9a7c30f63d170545aa70394aaebb6ca03706e31ab2b1272" is missing
WARN: replacing output file 6866075648..6870269952 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/5c8a/5c8ab7b6427d5ad152f34582a51de6663333e9a305b494bf31af0967a1d02b91" is missing
WARN: replacing output file 6870269952..6874464256 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/1360/136032f432a4e359f2c1045c252b23df2969a62713d9ad0270e463d3a7dfb7e2" is missing
WARN: replacing output file 6874464256..6878658560 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/9e95/9e95503b16da2549474d6e16595890a10eb6e60505b2959e8362fe8b9382fb62" is missing
WARN: replacing output file 6878658560..6882852864 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/a51e/a51eda168115e2226fda51c016df7d769e14860732e003223767968b50a0864f" is missing
WARN: replacing output file 6882852864..6887047168 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/f64c/f64cebebdf83f53ebe46e73f24f695e9a396f562cedc891b4df60722367c7693" is missing
WARN: replacing output file 6887047168..6891241472 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/c5fa/c5fa0c1d1f6aa3b965d3a9016740fc84df00519727c62d9ebcab15ef26f424e0" is missing
WARN: replacing output file 6891241472..6895435776 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/2e4f/2e4fb8763091da83436c2e58ec27ccaf1b8e191745f6761fe6d44b52ae1b154b" is missing
WARN: replacing output file 6895435776..6899630080 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/023c/023c000af1c670f1a034a78ec08ac104e33f03ee1f8f6a1f79dd82d23da83cae" is missing
WARN: replacing output file 6899630080..6903824384 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/5b91/5b912303b368ed38edd47e944ae3148e4100bacf698afaff615f34e3cd0798f7" is missing
WARN: replacing output file 6903824384..6908018688 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/6052/605211854a760db5e4975417d6d6322b8fd46ac44a76605b894e0673b8be49e1" is missing
WARN: replacing output file 6908018688..6912212992 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/0fb4/0fb4670f96993bf841e4b75413a128f9f15cf4c9ff1995beb5ed80f3ff05a3c2" is missing
WARN: replacing output file 6912212992..6916407296 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/db9a/db9a0f0962442c95924aff796e34c6f5c91828a0a74dc581406c1da0d3b58e33" is missing
WARN: replacing output file 6916407296..6920601600 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/c895/c895b46cfecf22ff766d3923404c62200316fb2b74f0c569b29c7a1e31b9ddaf" is missing
WARN: replacing output file 6920601600..6924795904 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/3475/3475d0a8d9d5dc150aaf1785baacf73f546549ae07c7491e1785942bd17813a4" is missing
WARN: replacing output file 6924795904..6928990208 with '\0'
WARN: chunk "/backup/proxmox/server1/.chunks/10ba/10ba2501cfa6bb1d2bbf9e96aa610f94f044345018455e7aa4dcc2986d57502a" is missing
WARN: replacing output file 6928990208..6933184512 with '\0'

 

[fabian]

is that an excerpt or the full output? you can pipe the output to a file and then either count the chunks or add up the segments.. but if a lot of chunks are missing, chances for recovery become slim..

 

[khangnch]

Hi.

The full output is in the attachment below. Please help us check if we are capable of recovering the old data.

Thank you very much.

 

[fabian]

well 8GB are missing, I don't know how big the file was (at least 16GB according to the output). but that likely means a plain attempt at recovery won't get you far, since too much of the file system is gone. depending on what data was on it that you want to recover, something like Photorec might restore at least some files.

 

[khangnch]

Hi.

The backup file is totally 20GB. We have mounted the output file to a new VM but when we execute ls -la, it shows nothing in the output.