Overwritten harddrive - Any hope of rescue?

[nicho1ab]

The ProxMox install CD was left in one of my ProxMox 1.3 machines (the master machine in a cluster). The machine was rebooted and I believe that ProxMox was partially re-installed on the machine.

The machine now boots into ProxMox from the HD, but gives the following errors after displaying the "Welcome to the ProxMox Virtual Environment. Please use your web browser to configure this server" banner display.

virtual1 login: kvm: 4586: cpu0 kcm_set_msr_common: MSR_IA32_MC0_STATUS 0x0 nop

ata2: timeout waiting for ADMA IDLE, stat=0x440
ata2: timeout wating for ADMA LEGACY, stat=0x440

The machine then reboots. I am able to briefly connect by SSH to this machine during the small window after SSHD starts, but before the machine reboots and can see that the qcow file for the one virtual machine I need to rescue is still on the disk. Is there any way to make this machine stop loading after it has loaded SSHD so that I can rescue the qcow file? Does anyone have any suggestions for recovering after this problem?

Thanks,

Andrew

 

[dietmar]

Proxmox does not reintall itself. So what exactly happened?

 

[dietmar]

Maybe the harddisk is damaged?

 

[udo]

Hi Nich1ab,
you can look at the harddisk when you boot a rescue-cdrom.

Something like http://grml.deb.at/grml64_2009.05.iso.torrent
Boot from the CD, activate the lvm:

# vgscan
# lvscan
# lvdisplay
# lvchange -a y /dev/pve/data
# lvchange -a y /dev/pve/root
# mount /dev/pve/root /mnt
# mount /dev/pve/data /mnt/var/lib/vz

copy all what you need with scp over the net, or on a second disk.

Good luck
Udo

 

[Omny_Devi]

As long as you don't write new data to the disk the chances of recovery are usually pretty decent. Don't install anything new.

The best way usually is to take the drive out and toss it in another machine or a dock with usb/esata/fireware (something) and load up the OS with some kind of recovery console. I have 4 USB devices with different versions of system recovery tools, (system rescue cd, knoppix, etc) which is pretty effective.

A quick google of "recover deleted linux data" found several promising results.

The main thing to do is not install anything new.

If the machine has raided arrays, it can get a bit more tricky.

Let me know how it goes!

For making good bootable usb devices I would look for

multicd.sh
unetbootin

Good luck mate

 

[nicho1ab]

@Dietmar - Thanks for the response. I thought that that ProxMox auto-installed if it booted from the CD. I'm glad to hear that it doesn't. That makes recovering easier.

@Udo - Great instructions! That is exactly what I needed. I didn't know how to mount an LVM file system, so I couldn't see anything in the file system, which is why I thought it had been overwritten. After following your instructions I've been able to see the VMs from this disk and am in the process of copying them over to another ProxMox installation.

@Omni_Devi - I did disconnect the drive as soon as I realized that something odd was afoot, but it now appears that it was my lack of understanding of LVM, rather than actual data loss that was preventing me from finding the data I need.

Thanks to all who responded. I'll post an update on how this works later today.

Andrew

 

[nicho1ab]

I was able to use UDO's instructions and was able to get the qcow2 file for this Windows 2003 VM. I've moved it to a backup ProxMox machine with the hopes that it was just going to show up in the list of VMs after a reboot, but that was not the case (that's not a complaint, it was just overly wishful thinking). I'm trying to figure out how to restore the VM now to backup ProxMox machine, but I didn't see any instructions in the HowTo section. I spent a few minutes scanning the forums for similar issues with resolutions and haven't come up with anything yet.

I'm going to keep looking, but is there a particular utility that can do this that I should be searching for?

Thanks again to everyone who offered solutions and for the speed with which you offered them.

Andrew

 

[udo]

Hi Andrew,
you need also the vm-settings e.g.

 /etc/qemu-server/106.conf

or do i misunderstand something?

Udo

 

[nicho1ab]

Ahh, OK. I hadn't gotten the conf file. That's exactly what I needed. It's now listed in the list of virtual machines. When I try to start it, it says that it is started in the ProxMox web interface. However, when I pull up the VNC console it says that the hard disk is not a bootable disk.

I made a backup copy of the qcow2 file and tried to convert it to raw to see if that would help at all, but when I run the following command:

qemu-img convert ncdev2-copy.qcow2 ncdev2-copy.raw

I get the following error:

qemu-img: error while reading

To transfer the qcow2 file from the failed server I used FTP and I think that it may have transferred in ASCII mode. Would that corrupt the qcow2 file possibly? Should I have transferred in binary?

Thanks,

Andrew

 

[dietmar]

To transfer the qcow2 file from the failed server I used FTP and I think that it may have transferred in ASCII mode. Would that corrupt the qcow2 file possibly? Should I have transferred in binary?

ASCII mode only transfer 7bit, so that corrupt the qcow file by sure. You should use binary mode instead.