Recover from a dead disk

[paulmorabi]

Hi everyone,

My SSD died on me a few days ago. Unfortunately, this had my Proxmox root partition and also several VM's. I also have two other disks which did not die and still seem to be fine. I'm replacing my SSD now so if I re-install on the new SSD, will I be able to recover the volumes and VM's I had on the other disks? Will Proxmox auto recognize them or do I need to do something to add them back in?

Thanks!

 

[fiona]

Hi,
the volumes for the VMs should be on the other disks, but you will have to recreate the VM and PVE configurations. First of all, you'll need to re-add the storages that were on the other disks. If you don't have any backup of /etc/pve, the fastest way to recover the VMs is probably to create a dummy VM with the same ID as the old one, and with the correct settings, except you don't need to select any install medium and can use a small hard disk which you can remove afterwards. Then you can use qm rescan --vmid <VMID> and the old disks of the VM with that ID will be added as unused disks to the configuration. Finally, attach the disks and select the correct boot disk.

 

[paulmorabi]

Thanks, I'll give this a try. I saw when Proxmox was booting it identified the other disks so once I recreate the vm, I should be able to also add the other volumes still there too?

 

[fiona]

Thanks, I'll give this a try. I saw when Proxmox was booting it identified the other disks so once I recreate the vm, I should be able to also add the other volumes still there too?

Of course you have to first mount/add the other disks to your host and create storage entries for them in PVE. qm rescan --vmid <VMID> will scan all configured storages for virtual disks with names vm-<VMID>-<DISKNAME> and add entries for them to the configuration file of the VM with that ID.

 

[paulmorabi]

I can see when I boot that Proxmox sees the other disks and they're listed on the web admin panel (in LVM and LVM Thin). However, in the left navigation I can only see local and local-lvm but not my two extra disks (called shared and ssd2).

I added a dummy VM and did a 'qm rescan' as you suggest but as the vdisk images are on ssd2, there is no results.

I know I'm missing something simple here but not sure what.

 

[paulmorabi]

I have some more data.

root@pve:~# lsblk
NAME                          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                             8:0    0 465.8G  0 disk
├─ssd2-ssd2_tmeta             253:11   0   4.7G  0 lvm 
│ └─ssd2-ssd2-tpool           253:13   0 456.3G  0 lvm 
│   ├─ssd2-ssd2               253:14   0 456.3G  0 lvm 
│   ├─ssd2-vm--103--disk--0   253:15   0   200G  0 lvm 
│   ├─ssd2-vm--103--disk--1   253:16   0     4M  0 lvm 
│   └─ssd2-vm--101--disk--0   253:17   0   128G  0 lvm 
└─ssd2-ssd2_tdata             253:12   0 456.3G  0 lvm 
  └─ssd2-ssd2-tpool           253:13   0 456.3G  0 lvm 
    ├─ssd2-ssd2               253:14   0 456.3G  0 lvm 
    ├─ssd2-vm--103--disk--0   253:15   0   200G  0 lvm 
    ├─ssd2-vm--103--disk--1   253:16   0     4M  0 lvm 
    └─ssd2-vm--101--disk--0   253:17   0   128G  0 lvm 
sdb                             8:16   0   1.8T  0 disk
├─shared-shared_tmeta         253:0    0   120M  0 lvm 
│ └─shared-shared-tpool       253:2    0   1.8T  0 lvm 
│   ├─shared-shared           253:3    0   1.8T  0 lvm 
│   ├─shared-vm--102--disk--0 253:4    0   800G  0 lvm 
│   └─shared-vm--100--disk--0 253:5    0   500G  0 lvm 
└─shared-shared_tdata         253:1    0   1.8T  0 lvm 
  └─shared-shared-tpool       253:2    0   1.8T  0 lvm 
    ├─shared-shared           253:3    0   1.8T  0 lvm 
    ├─shared-vm--102--disk--0 253:4    0   800G  0 lvm 
    └─shared-vm--100--disk--0 253:5    0   500G  0 lvm 
nvme0n1                       259:0    0 465.8G  0 disk
├─nvme0n1p1                   259:1    0  1007K  0 part
├─nvme0n1p2                   259:2    0   512M  0 part /boot/efi
└─nvme0n1p3                   259:3    0 465.3G  0 part
  ├─pve-swap                  253:6    0     8G  0 lvm  [SWAP]
  ├─pve-root                  253:7    0    96G  0 lvm  /
  ├─pve-data_tmeta            253:8    0   3.5G  0 lvm 
  │ └─pve-data-tpool          253:10   0 338.4G  0 lvm 
  │   └─pve-data              253:18   0 338.4G  0 lvm 
  └─pve-data_tdata            253:9    0 338.4G  0 lvm 
    └─pve-data-tpool          253:10   0 338.4G  0 lvm 
      └─pve-data              253:18   0 338.4G  0 lvm

The disks are definitely still there. When I make a VM with ID 103 and try to do a rescan, it does nothing:

root@pve:~# qm rescan --vmid=103
rescan volumes...
root@pve:~#

And when I try to list the images:

root@pve:~# pvesm list shared
Volid Format  Type      Size VMID
root@pve:~# pvesm list ssd2
Volid Format  Type      Size VMID
root@pve:~#

I'm stuck at this part because no matter what I try, I can't seem to get Proxmox to find the disks.

 

[fiona]

Sorry for the delay, I was on vacation. Did you add the old LVMs to PVE's storage configuration? E.g. via Datacenter > Storage > Add > LVM Thin in the GUI. Doing a qm rescan only looks at the configured storages and doesn't scan everything on the host.

If it still doesn't work after that, please share the output of the following commands:

cat /etc/pve/storage.cfg
lvs
vgs

 

[paulmorabi]

Thanks for the reply. I ended up using dd to get a raw image of the VM disks I wanted to restore and then re-initialized the drives in Proxmox and imported the images back in.

 

[bmcgonag]

@paulmorabi I know this is a bit old now, but can you define how you did your dd backups and imports please? I'm still learning Proxmox, but the same issue happened to me where the root partition disk of proxmox died, and I had to re-install, but now I have no idea how to get my VMs back. I still have the VMS, but also have the vmdump backups I was making on one of the drives. When I go to restore, I don't have a way to select that drive as the storage area through. I'm pretty bummed that there isn't a better way to 1 backup the Proxmox OS disk, and easily resotre from a new install in the UI. I have so much in it at this point, i hate to lose it all.

 

[Dunuin]

You can boot into a clonezilla usb stick and use it to backup your system disk (blocklevel backup of complete disk to a image file).

 

[paulmorabi]

@paulmorabi I know this is a bit old now, but can you define how you did your dd backups and imports please? I'm still learning Proxmox, but the same issue happened to me where the root partition disk of proxmox died, and I had to re-install, but now I have no idea how to get my VMs back. I still have the VMS, but also have the vmdump backups I was making on one of the drives. When I go to restore, I don't have a way to select that drive as the storage area through. I'm pretty bummed that there isn't a better way to 1 backup the Proxmox OS disk, and easily resotre from a new install in the UI. I have so much in it at this point, i hate to lose it all.

Sorry to hear you had a similar issue. Been there and done that. The good news is you should be able to recover everything if the images are still there. I don’t have the exact command I used any longer. However, you would use dd and refer to the lvm volume you want to backup. A guide for using dd with lvm is here. When you are restoring, if you still have the vm config then just copy or replicate the settings across to the new VM once you re-install Proxmox. From there, create a volume and use dd to restore the image to the new volume. Good luck with it.