VM disk deleted while running

[check-ict]

Hi,

I managed to delete a VM disk a long time ago, while it was still running. Now, 8 months later, I stopped and tried to start a VM but it won't start because the disk is missing.

8 months ago I did a move:
create full clone of drive virtio0 (ssd:100/vm-100-disk-0.qcow2)
Formatting '/mnt/pve/proxmox02_ssd02/images/100/vm-100-disk-0.qcow2'

create full clone of drive virtio0 (tempclone-proxmox:100/vm-100-disk-1.qcow2)
Formatting '/ssd/proxmox/images/100/vm-100-disk-0.qcow2',

After this move I deleted the unused disk that was present in the GUI after the move, but it shows the exact same disk as the one that was running!
update VM 100: -delete unused0
disk image '/ssd/proxmox/images/100/vm-100-disk-0.qcow2' does not exists
TASK OK

The task show OK but gives a error. Not sure why, but it seems to me that the VM disk was deleted this way.

When I check my snapshots, I see a difference between the hourly snapshot of the hour before I did a shutdown and one hour later when I tried to start the VM (and got the error).

ssd/proxmox@2021-07-07_20.00.01--2d 315M - 1.86T -
ssd/proxmox@2021-07-07_21.00.01--2d 164M - 1.57T -

This is the same disk size/data I'm missing. Is there a way to recover the disk in the old snapshot? The directory and file don't exist in the version of 20:00 so I can't recover the disk in a easy way. I tried both clone and ls the .zfs/snapshot folder.

Any tips on how to get this VM disk back? The back-ups only go back to 6 months, so we can't use backups (and it would be very old).

Everything was fine until the VM stopped after 20:00... The VM has been up all these months without any issues.

 

[mira]

If you've made the backups either with vzdump or to a PBS instance, you should be able to restore the backup.
Those run inside the QEMU process, so they still have the disk and its data available.

To test this, you could try restoring a newer backup to a different VMID and check the disks.

 

[check-ict]

We create ZFS snapshots and replicate them to 2 different systemens, with a retention of 6 months. Since the running disk was removed 8 months ago, we don't see the disk as a file anymore. We do see data change in the ZFS snapshots, so there is some data but it is not visible to us. This is also the case on the backup locations.

 

[check-ict]

We found the disk with zdb, by specifying the object:
zdb -ddddd hdd/test 384 | more
Object 384, 12k, 296GB real size, 500 GB disk size, 128k blk size, 512 dnsize

This is the file we are missing. We are trying to recover it now, but it's hard to find any information on this.

 

[check-ict]

We solved it by exporting the zdb information to a text file and filter out to only have this type of content:
5a0000 0:5e0867ba000:1bc00
5c0000 0:5dbcb34b400:1e000
5e0000 0:5d8a6580a00:14400
600000 0:674edfe3a00:14400
620000 0:5dba3caa600:19800
640000 0:5d964d0ea00:18c00
660000 0:a52f783b800:11a00
680000 0:674ec9b9e00:19800
6a0000 0:603e3f51400:1a400
6c0000 0:69bfa0f4400:10200
6e0000 0:4cf09264c00:17a00
700000 0:6035c267400:14400
720000 0:5db682d6400:17a00
740000 0:675ad227400:18c00
760000 0:51695d6e200:b400
780000 0:600d4f5e600:18000
7a0000 0:4cf0a19d000:e400
7c0000 0:676885c6400:14400
7e0000 0:89aa44c7000:16200

It was 4 million+ blocks, so that was a big file.

cat blocks.txt | while read line ; do zdb -R ssd/herstelzimbra $line >> /ssd/nobackup/zimbra-poging2.qcow2 ; done

This was really slow and it didnt handle "zeroes" and compression. We had to customize this script to our situation with compression enabled. Also we used screen to speed up the processen, running multiple processes. Each recovery atempt took a week with full speed and many failed, so we re-created the situation with a smaller test VM of 8GB, delete it while running, snapshot it and shutdown the VM. This smaller test was able to help us identify if our recovery scripts would work on the bigger file.