HELP - boot failed: not a bootable disk after failed backup

Discussion in 'Proxmox VE: Installation and configuration' started by vmunich, Sep 4, 2013.

  1. vmunich

    vmunich New Member

    Joined:
    Sep 4, 2013
    Messages:
    5
    Likes Received:
    0
    Ok, this is a .qcow2 Windows 2012 VM with VirtIO drivers. I have a very important database running in it, if I could only recover that DB it would be great.

    The VM was running fine but then I decided to transfer it to another node. This other node is not Proxmox based so I couldn't use the new Proxmox backup tool (which generates .vma files) so I used this command to make a backup:

    dd if=/var/lib/vz/images/402/vm-402-disk-1.raw | gzip | dd of=/var/lib/vz/images/402/vm-402-disk-1.gz bs=4096

    I forgot to shutdown the VM before I executed the command above, so I stopped the dd process. Then I decided to forget about DD and just use the Proxmox's backup tool instead because I found a tutorial teaching how to convert the .vma backups to .tgz. Since we can set the backup mode to "Stop", I didn't shut the VM down, and asked the backup tool to do it for me.

    The backup tool couldn't shutdown the VM by itself, so the backup failed.

    This is the log that Proxmox generated before my VM got screwed..:

    Sep 04 00:38:25 INFO: Starting Backup of VM 402 (qemu)
    Sep 04 00:38:25 INFO: status = running
    Sep 04 00:38:28 INFO: backup mode: stop
    Sep 04 00:38:28 INFO: bandwidth limit: 60000 KB/s
    Sep 04 00:38:28 INFO: ionice priority: 7
    Sep 04 00:38:28 INFO: stopping vm
    Sep 04 00:48:29 INFO: VM quit/powerdown failed - got timeout
    Sep 04 00:48:29 ERROR: Backup of VM 402 failed - command 'qm shutdown 402 --skiplock --keepActive --timeout 600' failed: exit code 255

    Then I opened the console of that VM and the VM was trying to boot but showing this message: boot failed: not a bootable disk after failed backup
    I restarted the VM several times, using write back, write back (unsafe), write through, no cache, and nothing works.

    I tried to mount the .qcow2 image but I failed. I converted the .qcow2 to .raw, downloaded ntfs-3g so debian could recognize the ntfs partitions and tried to mount the .raw image and it failed too. I tried running both Windows 2012 and Windows 7 isos and tried to repair the Windows installation but both failed too.

    Is there any way I could at least access the files inside the image??? I had a firebird database running in it, I just need to recover a 3gb .FDB file so I can restore it in a new VM.

    Help

    Thanks.

    Edit: I loaded the Windows 7 ISO and it shows that my drive has 120gb of free space in 120gb of unallocated space.. does it mean that I lost everything ???????? :/

    Linux disk utility shows this: http://d.pr/i/pZuF

    Is there anything I can do to try to recover the files??
     
    #1 vmunich, Sep 4, 2013
    Last edited: Sep 5, 2013
  2. vmunich

    vmunich New Member

    Joined:
    Sep 4, 2013
    Messages:
    5
    Likes Received:
    0
    Anyone???

    Bump
     
  3. symmcom

    symmcom Active Member

    Joined:
    Oct 28, 2012
    Messages:
    1,062
    Likes Received:
    16
    O man, soo sorry to hear what happened to your Windows :(

    I can feel your pain as i faced the similar issue once. But in my case after trying for 24 hours without any result i just restored the VM from a 3 days old backup. Although there was some data loss, but not a huge one since 2 of the 3 days were weekend.

    At later time i tried to recreate the same issue just to see what happens and i was able to recover about 94% of data by following solution:
    !!! MUST COPY THE DAMAGED VIRTUAL DISK SOMEWHERE TO PREVENT FURTHER WRITE ON IT. USE IT AS MASTER COPY !!! THE MORE YOU WILL TRY TO DO SOMETHING ON THE MASTER COPY IT WILL REDUCE THE CHANCE OF RESTORE !!!

    1. Create a fresh Windows VM.
    2. Add the damaged Virtual Disk as 2nd disk.
    3. Install program like Active Undelete.
    4. Create a new partition on the 2nd damaged virtual disk.
    5. Use Active Undelete to restore deleted/damaged partition/files.

    Let us know how this works out for you. Good Luck!!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. vmunich

    vmunich New Member

    Joined:
    Sep 4, 2013
    Messages:
    5
    Likes Received:
    0
    Oh my God thank you for giving me some hope man! I was already hopeless, tried testdisk but it didn't work, Im trying photorec now but it takes longer so while photorec tries to find anything I will try what you suggested me right now and see how it goes!!!
     
  5. vmunich

    vmunich New Member

    Joined:
    Sep 4, 2013
    Messages:
    5
    Likes Received:
    0
    It didn't work.. I tried sereval softwares, testdisk, photorec, loaded the damaged image on another windows VM and ran a bunch of softwares but they managed to recover only ~8gb..

    This is the log that Proxmox generated before my VM got screwed..:

    Sep 04 00:38:25 INFO: Starting Backup of VM 402 (qemu)
    Sep 04 00:38:25 INFO: status = running
    Sep 04 00:38:28 INFO: backup mode: stop
    Sep 04 00:38:28 INFO: bandwidth limit: 60000 KB/s
    Sep 04 00:38:28 INFO: ionice priority: 7
    Sep 04 00:38:28 INFO: stopping vm
    Sep 04 00:48:29 INFO: VM quit/powerdown failed - got timeout
    Sep 04 00:48:29 ERROR: Backup of VM 402 failed - command 'qm shutdown 402 --skiplock --keepActive --timeout 600' failed: exit code 255


    Can't you guys just leave the old backup tool (< Proxmox 2.2 )for the ones that would rather use it ?????????

    It would have been so much smooth if it was still available (for me) :(

    Sep 04 00:48:29 INFO: VM quit/powerdown failed - got timeout

    I noticed that more people got that same error while doing a backups.. But Proxmox's team insist that their .vma backups are much better and stable than the old way.. I'm not sure. Ins't Proxmox as reliable as it used to be before? :(

    OK I know that maybe the problem was that Proxmox wasn't able to shut the Windows VM down, but.. I still miss the old backup tool..
     
    #5 vmunich, Sep 5, 2013
    Last edited: Sep 5, 2013
  6. symmcom

    symmcom Active Member

    Joined:
    Oct 28, 2012
    Messages:
    1,062
    Likes Received:
    16
    Didyou try file restore or the entire partition restore? It must be volume/partition restore.

    Did you have cache enabled on the VM when everything was working? When cache enabled and vms are abruptly shut down, it causes unrecoverable error specially for windows. Same as raid cache that's why raid controller have battery backup. You are probably already familiar with this.

    Proxmox staff can correct me if I am wrong, but the new backup seems to be little faster and far stable, at least for all my dozen clusters.

    Sent from my SGH-T989D using Tapatalk 2
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. e100

    e100 Active Member
    Proxmox Subscriber

    Joined:
    Nov 6, 2010
    Messages:
    1,235
    Likes Received:
    24
    Thats what I was thinking too, maybe writeback was enabled.
    All of my disks are set to cache=none, we do not leave it at default, actually select cache=none just in case the default were to ever change.

    The other problem is trying to automatically shutdown windows, very unreliable in my experience. Without a clean shutdown problems will happen.
    I find that most of my Windows machines require two shutdown commands to actually shutdown, first one always fails, 2nd works 90% of the time.

    I prefer snapshot backups myself.
    I have only ever had one snapshot backup, fail to restore, but another backup of same VM did restore fine.
    We backup each VM 3X weekly and do test restores of the lastest backup of every VM each quarter, we have easily done over 800 snapshot backup restores with just one failure.
    No 2012 VMs yet, only 2003 and 2008
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice