rescue advice needed - recover info from VM - HowTo?

[iPanini]

Hi,
I'd like some advice of the more knowledgeable proxmox users around on the forum.
Here's what happened:
- some time, months ago, my single proxmox machine got halted due to an unplanned power interruption.
- again a while later I started receiving error messages in my mailbox: SMART error (OfflineUncorrectableSector) detected on host: pve-1
Due to a total lack of time this error was not taken care of.
- yesterday we did have a planned power interruption, so I used the GUI to properly shutdown pve-1
- then later, while trying to power up again, I of course received more errors, all pointing to a disk failure
You'll find a couple of screenshots below.

Now my question:
- it seems LVM does not come online (due to the disk failure)
=> TASK ERROR: activating LV 'pve/data' failed: Activation of logical volume pve/data is prohibited while logical volume pve/data_tmeta is active.
- after another reboot, I was able to log in on the terminal, pve-1 did have no network connection, so I could only work on the physical machine itself
- I tried to run some SMART commands to check
- then after another reboot, and some more time, the network connection was reestablished, and I could again approach the GUI
- the single one VM that I'd like to rescue is my Home Assistant install, doing an ls -la in terminal, shows the presence of haos_ova-11.2.qcow2, like this:

-rw-r--r-- 1 root root 1001259008 Dec 30 2023 haos_ova-11.2.qcow2

What can I do to rescue my HA VM?
- is the qcow2 file the VM or?

Some thoughts / idea's:
- I was already playing with the idea of setting up a 2nd proxmox installation, I'm still on Proxmox 8.2.7, and then maybe try to move the VM over to the new PVE-1?

Looking forward to any help, tips, suggestions & idea's & thanks for helping out!

 

[bbgeek17]

Hi @iPanini ,
The qcow you found is likely NOT your VM. It seems to be located in your home directory. More than likely it is the installation download you got originally. The 2003 date of the file is another confirmation.

The LVM error message indicates some metadata corruption that often happens in unplanned power down with consumer hardware. There are many guides online that can walk you through the potential recovery steps.

One of your options is to duplicate the disk (dd or similar) prior to attempting recovery.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[iPanini]

There are many guides online that can walk you through the potential recovery steps.

Hi bbgeek17, thanks for replying.
I've done a quick search and can see some light at the end of the tunnel ;-)
Are you aware of any such guide related to Proxmox?
I would prefer to find a guide related to Proxmox, as one of the things I found relates to Thin provisioning, and frankly my knowledge of LVM is limited.

 

[bbgeek17]

Are you aware of any such guide related to Proxmox?
I would prefer to find a guide related to Proxmox, as one of the things I found relates to Thin provisioning, and frankly my knowledge of LVM is limited.

Proxmox is based on Debian Linux with a curated Ubuntu Kernel. The LVM is part of standard Linux implementation. There are no Proxmox-specific steps when dealing with LVM.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[iPanini]

The LVM error message indicates some metadata corruption that often happens in unplanned power down with consumer hardware. There are many guides online that can walk you through the potential recovery steps.

I'd like some further guidance please:
I'm trying to follow instructions from Hetzner, here.
So to start, I want to login using my Proxmox usb installer stick. FYI, I have the installer on a Ventoy stick.
- connected the usb stick to PVE
- booted into BIOS
- adjusted boot options to be able to boot from stick
- reboot
- then quite some different screens (several photo's available) and what I think was a verbose boot with lots of information scrolling by
- selected Advanced Options
- selected Rescue Boot

- saw message: error: unknown filesystem, Press any key to continue
- then boot process continued verbose
- I could clearly see that the boot process also identified the diskdrive error
- I would have then expected to be dropped into a terminal, but..
- ended up with the usual Proxmox message: Welcome to Proxmox .. connect to: https://ip-address:8006/
- so it seems that somewhere the boot process from the usb-installer stick has started the installed Proxmox instance instead of staying within the boundaries of the usb-stick boot process?

Question:
- How can I force Proxmox login to just login to terminal without starting PVE? I assume this should be possible using the installer usb-stick?
- Or do I need to create another bootable (Debian) stick and try using a live distro?

Thanks for helping out!

 

[bbgeek17]

Perhaps check this thread and the one referred in it:

M

Thread 'SOLVED: Proxmox 8.0 Rescue Boot help needed'

Aug 30, 2023

Hi, on one of my Proxmox VE servers, I accidentally did something stupid. I forgot to uninstall the nVidia drivers after upgrading from 7 to 8. After I rebooted the system, all I got was the following screen

Booting `Proxmox VE GNU/Linux'

Loading Linux 6.2.16-8-pve ...

I tried to press and hold <shift>, or <esc> to get to the grub menu but no luck. So I thought I would try using the Proxmox on my USB stick and get to the system via Rescue Boot. However, upon running it, I got the following error

error: no such device: rpool.
ERROR: unable to find boot disk automatically...

• matrix1999
• Replies: 4
• Forum: Proxmox VE: Installation and configuration

Creating a vanilla Debian/Ubuntu rescue disk is also an option. There is nothing PVE specific in what you are trying to do.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox