A bunch of VM's drives went blank

[raven975]

So I am running Proxmox 8.0.3. I moved over from ESXi to Prox about a year ago. I migrated all my vmdk's and converted them to raw format and was able to get my servers up and running. Last week, for an unknown reason a bunch of my servers went down. When trying to boot them back up it said there was no boot drive available. The raw drives were still connected but the vm was not booting from them. I used a live cd and it shows the drive as unallocated and ready to be formatted. I am not sure what happened to cause so many raw drives to lose all the contained data or at least lose the MBR to not allow data to be seen anymore. Has anyone experienced anything like this and is it possible to recover any data from these?

As note, I have restored most the servers from various backup repo's that I have so things are back up and running but this is more for educational. I do still have "dead servers" that I can experiment with to see if I can find a solution.

 

[Dunuin]

So I am running Proxmox 8.0.3. I moved over from ESXi to Prox about a year ago. I migrated all my vmdk's and converted them to raw format and was able to get my servers up and running. Last week, for an unknown reason a bunch of my servers went down. When trying to boot them back up it said there was no boot drive available. The raw drives were still connected but the vm was not booting from them. I used a live cd and it shows the drive as unallocated and ready to be formatted. I am not sure what happened to cause so many raw drives to lose all the contained data or at least lose the MBR to not allow data to be seen anymore. Has anyone experienced anything like this and is it possible to recover any data from these?

You never updated PVE after installing it a year ago and there is a known bug that might wipe the first sector of the MBR (and with damaged MBR there will be no partitions shown like you described) of virtual disks when using SATA as protocol. I bet your VMs are using SATA despite using virtio SCSI would be best practice. That bug got patched some months ago with PVE 8.1 but as you never updated your PVE you probably still ran into this bug.

See: https://bugzilla.proxmox.com/show_bug.cgi?id=2874

 

[raven975]

Good to know. So is the data gone then? Also I will be spending the evening updating my two proxmox to the latest version.

 

[Dunuin]

You might want to search this forum for other threads. You are not the first one with a corrupted MBR partition table caused by this bug and I think I remember people fixing this by cloning the partition table or something similar.

I also edited my post above in case you didn't see the link to the bugtracker ticket for this bug.

 

[raven975]

if its fixing the MBR and it can be done similarly to other devices and not specific to Proxmox, that at least gives me a direction to go in. Thank you for this. I have looked for a week and couldn't find this on any of the forums.

 

[raven975]

I found the answer:
https://bugzilla.proxmox.com/show_bug.cgi?id=2874#c38

For Linux VMs
Setup VM with debian for example "VM debian repair"
Install "testdisk" tool
Move your damaged disk to this VM
Repair table with testdisk
Use "fsch" to repair your FS (this is impotant)
Then repair grub with grub2-install (there is a lot of different ways to repair your grub so i add to my "VM debian repair" another disk with same but empty VM like damage one (for me it is CentOS 7.9), copy /boot and run grub2-install to prevent some bugs if you use Debian tools)



For windwos VMs
Setup VM with any modern wiondwos (10 for instance)
Setup Disk Genius tool (not free)
Repair disk tables
Check and repair fs for all disks (NTFS)
Add DVD with same windwos as your damaged one
Repair bootloader with
bootrec /rebuildbcd
bootrec /fixmbr


Thank you again for directing me to this post!