Disaster recovery

[stsinc]

I am in a difficult situation. Here is what happened:

1. I needed to get more space on each of the boot partition of my 5 Proxmox v7 nodes
2. So I foolishly applied a cleaning script that has made ALL my 5 nodes unbootable
3. Basically, when I boot each I get the Memory test and NO Proxmox entry (when applying `proxmox-boot-tool status` I get `proxmox-boot-uuids` not found)

I have made image backups of each of the boot SSDs of the 5 nodes.
I can mount the LVMs inside each (except for the `data` LV which has no path) and I know the data are there, in `ext4` format.

So:

1. is there a way I can reinstall Proxmox v7 on each node without erasing the `data` partition where the CTs and VMs are stored?
2. If not possible, is there a way I can recover these CTs and VMs and "reinject" them into a brand new Proxmox install?

Could you send me ponters to the solution I could use?

Thanks
Stephen

 

[flames]

Hi,
show us your "cleaning" script. i would guess it removed the active pve kernel. you could reinstall it by booting from a debian rescue mode, mount the proxmox disk, add proxmox repositories and install with apt.

Your questions:
1. no, if you can't boot into that ssd and use apt. another way would be to boot rescue from debian iso/cd, mount the ssd and reinstall with apt from there.
2. if you install proxmox to new ssds, you can find the configs including containers and VMs in /etc/pve, if you are running ceph, also /etc/ceph
2.1 before you reinstall on the same ssds, check, that your images are really working and you can extract the configs from there. also you could just restore images, but best on new ssd.

 

[stsinc]

Hi, and thank you for your prompt response.

• I will try and find the "cleaning script" which have deleted from all my computers since ;o)) BTW, I found it in a user's response on this very forum.
• I am convinced #1 solution is the right one. Could you give me the full procedure please or direct me to an existing link?

Thanks again,
Stephen

 

[flames]

i need to know more about your setup. how did you originally installed proxmox? zfs-boot or default lvm? are your vms on a local/local-lvm/local-zfs storage (same disk as OS) or is there a dedicated storage as separate zfs, ceph cluster or something... the more info you can tell, the better we can help

what exactly happens, when you boot from the current ssd? screenshot?

 

[stsinc]

Here are my answers @flames :

1. default lvm
2. same disk as OS
3. no ceph, zfs, etc. Plain simple install on SSDs

When I boot, the GRUB does not display any Proxmox entry any longer, just different entries of MemoryTest (which fails, BTW despite the fact I checked the memory with Lenovo's own tools and it is fine).
I then tried to use the DEBUG install mode and could execute `proxmox-boot-tool status`

 

[flames]

1. Basically, when I boot each I get the Memory test and NO Proxmox entry (when applying `proxmox-boot-tool status` I get `proxmox-boot-uuids` not found)

did you already booted into a rescue disk and had a look into your ssd, if all the data is still there? especially /etc/pve still contains all the configs

 

[stsinc]

Here is what I did:

1. I unmounted the SSDs from each of the physical machines and clonezilla-ed them each
2. Then I inserted them each in an enclosure and mounted them on my workstation using `vgscan` then `vgdisplay` `vgchange -ay` and so on. The report showed that every LV was fine.

Meaning: I had the `swap`, the `pve-root` and the `data` (which I was unable to mount but it is for yet another reason)

 

[flames]

sounds good so far.
missing boot-uuids means actually, that you are booting in legacy mode (grub), not uefi. if i remember correctly, proxmox-boot-tool is for uefi boot.
did you eventually changed something in the boot options in bios/eufi?

 

[stsinc]

No, I do not think so, stayed in mixed Legacy/UEFI mode with a preference for Legacy

 

[flames]

i think we only need to fix grub entries then. https://pve.proxmox.com/wiki/Recover_From_Grub_Failure

 

[flames]

if that does not work and pve kernel is also gone, you could mount pve-root again, chroot into it and "apt install proxmox-ve"

 

[stsinc]

OK, that is the procedure I was looking for.
I also forgot to tell you I tried a boot repair from the live ISO Boot Repair Disk and it did not work

 

[stsinc]

I will have breakfast (it is 7.30 AM here in NYC) and will try after.
THANKS !!

 

[flames]

time zones... i will come back here in 4-5 hours, afternoon at work currently.

 

[stsinc]

OK @flames , here is the latest update from the recovery front.

1. I boot from the Proxmox live install
2. I access the DEBUG mode
3. `exit` to go back to the shell
4. mount `/dev/pve/root`
5. but when I execute `apt install proxmox-ve` I get an error message saying it is not found.

The reason is that I am not connected to the Internet in that mode (despite the fact I am physically connected to my router).
And I did an `ip a` and my link is, in effect, down.

So my question is: how do I connect to the Internet from the debug mode?

 

[flames]

not only mount pve-root, but chroot into it as described here https://pve.proxmox.com/wiki/Recover_From_Grub_Failure
then apt update && apt install proxmox-ve

 

[stsinc]

Ah, I thought only mount it...

 

[flames]

when you chroot into the volume, your rescue system acts as if pve-root is its own boot volume but running from rescue kernel. so you can issue commands "like" you booted from the offline system (not everything works ofc., but yet it helps).

 

[stsinc]

I get it now.
The error I was also making was to have booted from a Ventoy based Proxmox. Redoing the whole process with a plain USB installed Proxmox installer.

 

[stsinc]

I am hitting a blocking point when I execute

mount /dev/sda1 /media/RESCUE/boot

I get an error message saying that sda1 already mounted or mount point busy