Recover Deleted VM

mitzaodx

New Member
Sep 1, 2013
7
0
1
Hello. I will get right to the point because I am panicking a little bit right now.
I had 2 VMs running FreeNAS. The first VM with the VID 103 was running FreeNAS 9.1 RC1 and the second VM with VID 105 running FreeNAS 9.1 RC2.
The ideea at the time was to move all of my data from the first VM to the second VM so I can delete the old FreeNAS. Yesterday I finished moving all my data (videos, music, all photos) to the VM with VID 105.
Today I wanted to remove the FreeNAS and I accidentally deleted the VM with VID 105 instead of 103 from the web interface.
After that I created a new VM running Ubuntu 13.04 to migrate all my data there because FreeNAS had some speed issues running as a VM.
So I created a new VM with VID 105. When I needed to access some of my photos, I realized the VM that I needed to delete was the wrong one.
As a total noob I started looking for a way to recover my files. I knew that I had 2 virtual disks created. One for the OS, 6 GB and one for data, 300 GB.
I found out that there are information stored about the disks here:
/etc/lvm/archive/hdd1000gb_00040-1393861768.vg

In the description of the file, it says:

description = "Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-105-disk-2'"

Code:
vm-105-disk-1 {
                        id = "HowMJY-CY67-2swc-d7u5-MkEb-wgfS-AaZf5D"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        tags = ["pve-vm-105"]
                        creation_host = "srv1-lnx"
                        creation_time = 1374978289      # 2013-07-28 05:24:49 +0300
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 1536     # 6 Gigabytes


                                type = "striped"
                                stripe_count = 1        # linear


                                stripes = [
                                        "pv0", 219136
                                ]
                        }
                }


                vm-105-disk-2 {
                        id = "xicJnd-Tm8g-dQy4-PgA2-6b9h-lgKH-mVBWzj"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        tags = ["pve-vm-105"]
                        creation_host = "srv1-lnx"
                        creation_time = 1374978329      # 2013-07-28 05:25:29 +0300
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 76800    # 300 Gigabytes


                                type = "striped"
                                stripe_count = 1        # linear


                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }

I had no backup of the data since I just finished moving it all in one place and I didn't need to run a full backup yet.
Is there a way I can restore those virtual disk?

I forgot to add that I have Proxmox version 3.0.23.
 
Last edited:
Never had to deal with something like this so not really sure what to advise.
Seems like starting here is a good step: http://pve.proxmox.com/wiki/Backup_and_restore_of_LVM_data

Do this, nothing!
Do not make new VMs, don't try to fix it.

Give me a few minutes and I will research this a bit hopefully I can figure out what commands to run that will provide the data necessary to determine if this can be fixed or not.
Maybe someone who has dealt with this will drop in too.
 
Never had to deal with something like this so not really sure what to advise.
Seems like starting here is a good step: http://pve.proxmox.com/wiki/Backup_and_restore_of_LVM_data

Do this, nothing!
Do not make new VMs, don't try to fix it.

Give me a few minutes and I will research this a bit hopefully I can figure out what commands to run that will provide the data necessary to determine if this can be fixed or not.
Maybe someone who has dealt with this will drop in too.

I didn't create any other VMs except the Ubuntu One. After I figured that I deleted the wrong VM I shutdown the Ubuntu VM and I deleted it. Nothing else was touched.
 
This seems like the best description of what you can do:
http://www.anchor.com.au/blog/2011/06/elegantly-reverting-unintentional-lvm-changes/

This will show what is restorable, please post the output here:
Code:
vgcfgrestore --list [COLOR=#333333]hdd1000gb[/COLOR][COLOR=#008000][FONT=Helvetica]
[/FONT][/COLOR]

Code:
  File:         /etc/lvm/archive/hdd1000gb_00027-25133061.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-110 --size 10485760k --name vm-110-disk-1 hdd1000gb'
  Backup Time:  Sun Aug 11 00:33:39 2013




  File:         /etc/lvm/archive/hdd1000gb_00028-636401040.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-110 --size 10485760k --name vm-110-disk-2 hdd1000gb'
  Backup Time:  Sun Aug 11 00:40:08 2013




  File:         /etc/lvm/archive/hdd1000gb_00029-560880805.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-110-disk-1'
  Backup Time:  Sun Aug 11 00:43:13 2013




  File:         /etc/lvm/archive/hdd1000gb_00030-201835069.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-111 --size 10485760k --name vm-111-disk-1 hdd1000gb'
  Backup Time:  Sun Aug 11 00:47:25 2013




  File:         /etc/lvm/archive/hdd1000gb_00031-2121733797.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-111 --size 10485760k --name vm-111-disk-2 hdd1000gb'
  Backup Time:  Sun Aug 11 01:03:46 2013




  File:         /etc/lvm/archive/hdd1000gb_00032-1764672392.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-111-disk-2'
  Backup Time:  Sun Aug 11 01:14:05 2013




  File:         /etc/lvm/archive/hdd1000gb_00033-1753785.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-111-disk-1'
  Backup Time:  Sun Aug 11 01:14:05 2013




  File:         /etc/lvm/archive/hdd1000gb_00034-1615499505.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-111 --size 33554432k --name vm-111-disk-1 hdd1000gb'
  Backup Time:  Sun Aug 11 01:14:56 2013




  File:         /etc/lvm/archive/hdd1000gb_00035-1409598243.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-111-disk-1'
  Backup Time:  Sun Aug 11 01:33:02 2013




  File:         /etc/lvm/archive/hdd1000gb_00036-665016760.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-111 --size 33554432k --name vm-111-disk-1 hdd1000gb'
  Backup Time:  Sun Aug 11 01:36:20 2013




  File:         /etc/lvm/archive/hdd1000gb_00037-1109813545.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-120 --size 10485760k --name vm-120-disk-1 hdd1000gb'
  Backup Time:  Sun Aug 11 13:05:02 2013




  File:         /etc/lvm/archive/hdd1000gb_00038-2136373485.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-110-disk-2'
  Backup Time:  Sun Aug 11 15:35:35 2013




  File:         /etc/lvm/archive/hdd1000gb_00039-237453448.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-120-disk-1'
  Backup Time:  Sun Aug 11 16:48:13 2013




  File:         /etc/lvm/archive/hdd1000gb_00040-1393861768.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-105-disk-2'
  Backup Time:  Sun Sep  1 13:00:40 2013




  File:         /etc/lvm/archive/hdd1000gb_00041-1100323822.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-105-disk-1'
  Backup Time:  Sun Sep  1 13:00:40 2013




  File:         /etc/lvm/archive/hdd1000gb_00042-985048105.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-105 --size 314572800k --name vm-105-disk-1 hdd1000gb'
  Backup Time:  Sun Sep  1 13:07:17 2013




  File:         /etc/lvm/archive/hdd1000gb_00043-1936228725.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvcreate -aly --addtag pve-vm-105 --size 33554432k --name vm-105-disk-2 hdd1000gb'
  Backup Time:  Sun Sep  1 13:08:36 2013




  File:         /etc/lvm/archive/hdd1000gb_00044-916344814.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-105-disk-1'
  Backup Time:  Sun Sep  1 15:31:46 2013




  File:         /etc/lvm/archive/hdd1000gb_00045-694075388.vg
  VG name:      hdd1000gb
  Description:  Created *before* executing '/sbin/lvremove -f hdd1000gb/vm-105-disk-2'
  Backup Time:  Sun Sep  1 15:31:46 2013




  File:         /etc/lvm/backup/hdd1000gb
  VG name:      hdd1000gb
  Description:  Created *after* executing '/sbin/lvremove -f hdd1000gb/vm-105-disk-2'
  Backup Time:  Sun Sep  1 15:31:46 2013
 
I suspect your overwrote the data when you created the new ubuntu vm, but maybe LVM put the ubuntu disks in some other location.
If your data was on disk-2, it is possible it might be accessible, if it is some of it is very likely corrupted.

My only suggestion is to:
#1. backup everything in case this causes more problems, don't make a bad situation worse.
#2. Restore the lvm data using the backup files
vgcfgrestore --file /etc/lvm/archive/hdd1000gb_00040-1393861768.vg

As I mentioned before I've never had to deal with this sort of situation so my advice is simply what I would do if I were in your shoes.
No idea if it will help or make things worse, might even damage your working VMs, so don't forget to backup first.

If possible I would copy the whole disk using dd and try recovering your data from the copy.
Imagine if someone posts tomorrow saying "if you had done XYZ you could get your data but now that you did ABC you cannot"

Once you run the restore command, you can try to mount the disk directly in the proxmox host to see if your data is there, or not, using a utility "kpartx"
This looks like a good description on how to do that:
https://www.soljerome.com/blog/2011/05/07/using-kpartx-to-read-lvm-volumes/
 
I suspect your overwrote the data when you created the new ubuntu vm, but maybe LVM put the ubuntu disks in some other location.
If your data was on disk-2, it is possible it might be accessible, if it is some of it is very likely corrupted.

My only suggestion is to:
#1. backup everything in case this causes more problems, don't make a bad situation worse.
#2. Restore the lvm data using the backup files
vgcfgrestore --file /etc/lvm/archive/hdd1000gb_00040-1393861768.vg

As I mentioned before I've never had to deal with this sort of situation so my advice is simply what I would do if I were in your shoes.
No idea if it will help or make things worse, might even damage your working VMs, so don't forget to backup first.

If possible I would copy the whole disk using dd and try recovering your data from the copy.
Imagine if someone posts tomorrow saying "if you had done XYZ you could get your data but now that you did ABC you cannot"

Once you run the restore command, you can try to mount the disk directly in the proxmox host to see if your data is there, or not, using a utility "kpartx"
This looks like a good description on how to do that:
https://www.soljerome.com/blog/2011/05/07/using-kpartx-to-read-lvm-volumes/



Just a question to the OP. If you were running Freenas, was the data in question stored on ZFS? If so, were snapshots enabled? You might be able to roll back and recover if so.
 
Just a question to the OP. If you were running Freenas, was the data in question stored on ZFS? If so, were snapshots enabled? You might be able to roll back and recover if so.

Data on both freenas VMs were on ZFS. Snapshots were enabled on the first one and I didn't manage to get them recovered, that's why I installed the second one and moved all my data there.
I was going to move it from the second one with Freenas RC2 to ubuntu and get it backed up manually with rsync because it still didn't work so I wanted to drop the whole freenas thing.
Tonight I am going to try and recover and I will come back and post the results.
 
I suspect your overwrote the data when you created the new ubuntu vm, but maybe LVM put the ubuntu disks in some other location.
If your data was on disk-2, it is possible it might be accessible, if it is some of it is very likely corrupted.

My only suggestion is to:
#1. backup everything in case this causes more problems, don't make a bad situation worse.
#2. Restore the lvm data using the backup files
vgcfgrestore --file /etc/lvm/archive/hdd1000gb_00040-1393861768.vg

As I mentioned before I've never had to deal with this sort of situation so my advice is simply what I would do if I were in your shoes.
No idea if it will help or make things worse, might even damage your working VMs, so don't forget to backup first.

If possible I would copy the whole disk using dd and try recovering your data from the copy.
Imagine if someone posts tomorrow saying "if you had done XYZ you could get your data but now that you did ABC you cannot"

Once you run the restore command, you can try to mount the disk directly in the proxmox host to see if your data is there, or not, using a utility "kpartx"
This looks like a good description on how to do that:
https://www.soljerome.com/blog/2011/05/07/using-kpartx-to-read-lvm-volumes/

I have backed up my other VMs and entered the command:

Code:
vgcfgrestore --file [COLOR=#333333]/etc/lvm/archive/hdd1000gb_00040-1393861768.vg[/COLOR]

I've got this output:

Code:
 Please specify a *single* volume group to restore.

I did a google search and found out that I had to add the volume group name to the command so I executed this:

Code:
vgcfgrestore --file /etc/lvm/archive/hdd1000gb_00040-1393861768.vg [B]hdd1000gb[/B]

And the output was this:

Code:
Restored volume group hdd1000gb


I checked in the Proxmox interface and i can see the disks.
I have to do a presentation in the morning and can't stay and look for more answers. If you can find out something about recovering the VM configuration it would be great :).
I know that the VM was setup with a KVM64 processor if that would help.

I will be able to enter from work and see if I can continue.

Thank you for your time for helping me.
 
Last edited:
Do you see your missing disks if you run the following command?

sudo lvs

A disk is using the following naming convention: vm-<vmid>-disk-<disk number>
 
if disk is in raw format then each lvm will consists of one or more disk partitions so do the following:

1) Identified disk image you what to restore (lvdisplay and choose the vm-<vmid>-disk-<disk number> of interest)
2) Activate disk image (Assuming /dev/some_vg/vm-100-disk-1): lvchange -a y /dev/some_vg/vm-100-disk-1
3) List partitions: fdisk -l /dev/some_vg/vm-100-disk-1
4) Assuming:
Code:
Disk /dev/some_vg/vm-100-disk-1: 4303 MB, 4303355904 bytes
255 heads, 63 sectors/track, 523 cylinders, total 8404992 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 262144 bytes
Disk identifier: 0x000601ad


                      Device Boot      Start         End      Blocks   Id  System
/dev/some_vg/vm-100-disk-1p1   *        2048     7938047     3968000   83  Linux
/dev/some_vg/vm-100-disk-1p2         7940094     8386559      223233    5  Extended
/dev/some_vg/vm-100-disk-1p5         7940096     8386559      223232   82  Linux swap / Solaris
Partition of interest will therefore be /dev/some_vg/vm-100-disk-1p1
losetup /dev/loop0 /dev/some_vg/vm-100-disk-1
5) apt-get install kpartx
6) kpartx -a /dev/loop0
7) mkdir /mnt/old_disk
8) mount /dev/mapper/loop0p1 /mnt/old_disk
9) browse /mnt/old_disk and fetch files of interest
10) When finished: umount /mnt/old_disk
11) kpartx -d /dev/loop0
12) losetup -d /dev/loop0
13) lvchange -a n /dev/some_vg/vm-100-disk-1

Thanks it!
 
if disk is in raw format then each lvm will consists of one or more disk partitions so do the following:

1) Identified disk image you what to restore (lvdisplay and choose the vm-<vmid>-disk-<disk number> of interest)
2) Activate disk image (Assuming /dev/some_vg/vm-100-disk-1): lvchange -a y /dev/some_vg/vm-100-disk-1
3) List partitions: fdisk -l /dev/some_vg/vm-100-disk-1
4) Assuming:
Code:
Disk /dev/some_vg/vm-100-disk-1: 4303 MB, 4303355904 bytes
255 heads, 63 sectors/track, 523 cylinders, total 8404992 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 262144 bytes
Disk identifier: 0x000601ad


                      Device Boot      Start         End      Blocks   Id  System
/dev/some_vg/vm-100-disk-1p1   *        2048     7938047     3968000   83  Linux
/dev/some_vg/vm-100-disk-1p2         7940094     8386559      223233    5  Extended
/dev/some_vg/vm-100-disk-1p5         7940096     8386559      223232   82  Linux swap / Solaris
Partition of interest will therefore be /dev/some_vg/vm-100-disk-1p1
losetup /dev/loop0 /dev/some_vg/vm-100-disk-1
5) apt-get install kpartx
6) kpartx -a /dev/loop0
7) mkdir /mnt/old_disk
8) mount /dev/mapper/loop0p1 /mnt/old_disk
9) browse /mnt/old_disk and fetch files of interest
10) When finished: umount /mnt/old_disk
11) kpartx -d /dev/loop0
12) losetup -d /dev/loop0
13) lvchange -a n /dev/some_vg/vm-100-disk-1

Thanks it!

Thank you for the tutorial but what it has been recovered was the ubuntu partition because it was the same name as the VM. I tried restoring different volume groups but it didn't help.
I will try with other software to recover the files from my windows machine because at one point they were there :).

Thanks for all the support.
 
Hi,
Please, can i have the detailled solution, we have a vm that has been deleted, and we wanna restore it ,
Please help
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!