[SOLVED] Recover From Grub Failure: Can't find LVM

[rust3d]

I can't upgrade https://pve.proxmox.com/wiki/Recover_From_Grub_Failure this wiki entry so I will add this update for anyone who needs it with EFI.

After changing out a GPU in my computer I came across this grub rescue screen.

error: disk `lvmid/[LVMID]' not found.

• grub rescue >

I needed to use a 64 bit version of Ubuntu or Debian live usb disk to fix it.

Boot into Ubuntu/Debian off the usb. We do not want to install Ubuntu/Debian, just run it live off the ISO/DVD/USB. Warning: Debian Live goes to sleep after 20 minutes inactivity if you for some reason have to take a break.

First We need to activate LVM and mount the the root partition that is inside the LVM container.

• sudo vgscan
• sudo vgchange -ay

Mount all the filesystems that are already there so we can upgrade/install grub. Your paths may vary depending on your drive configuration.

• sudo mkdir /media/RESCUE
• sudo mount /dev/pve/root /media/RESCUE/
• sudo mount /dev/[partition of efi, for me nvme0n1p2] /media/RESCUE/boot/efi
• sudo mount -t proc proc /media/RESCUE/proc
• sudo mount -t sysfs sys /media/RESCUE/sys
• sudo mount -o bind /dev /media/RESCUE/dev
• sudo mount -o bind /run /media/RESCUE/run

Chroot into your proxmox install.

• chroot /media/RESCUE

Since my UUID changed on hardware I had to extend my root

• lvextend -L +1g /dev/pve/root
• resize2fs /dev/pve/root

You could also rename the vg and rename it back instead of the last step.

Then update grub and install it.

• update-grub
• grub-install

If there are no error messages, you should be able to reboot now.

here is the bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=987008

 

[cr2]

Exactly the same happened to me today, but I was flashing a new BIOS to my server after a twin server failed yesterday due to BIOS corruption. I decided to update all my servers and on one I got this error. Trying your instructions now, thanks for posting.

Update:
It worked!
For newbies like me, to find your EFI partition run the command lsblk and find a partition (the column type says 'PART') with a size of approximately 512MB, in my case it was /dev/sdb2

Finally, I don't understand why the resize is needed, I guess it is a quick way to shake the nest and tell grub things have change, but without the resize it didn't work.

 

[rust3d]

Exactly the same happened to me today, but I was flashing a new BIOS to my server after a twin server failed yesterday due to BIOS corruption. I decided to update all my servers and on one I got this error. Trying your instructions now, thanks for posting.

Update:
It worked!
For newbies like me, to find your EFI partition run the command lsblk and find a partition (the column type says 'PART') with a size of approximately 512MB, in my case it was /dev/sdb2

Finally, I don't understand why the resize is needed, I guess it is a quick way to shake the nest and tell grub things have change, but without the resize it didn't work.

The resize or rename forces an update in the uuid. The problem is that the uuid is not getting updated so it's an easy way for force it.

 

[RajatSharma]

I can't upgrade https://pve.proxmox.com/wiki/Recover_From_Grub_Failure this wiki entry so I will add this update for anyone who needs it with EFI.

After changing out a GPU in my computer I came across this grub rescue screen. View attachment 46189

error: disk `lvmid/[LVMID]' not found.

• grub rescue >

I needed to use a 64 bit version of Ubuntu or Debian live usb disk to fix it.

Boot into Ubuntu/Debian off the usb. We do not want to install Ubuntu/Debian, just run it live off the ISO/DVD/USB. Warning: Debian Live goes to sleep after 20 minutes inactivity if you for some reason have to take a break.

First We need to activate LVM and mount the the root partition that is inside the LVM container.

• sudo vgscan
• sudo vgchange -ay

Mount all the filesystems that are already there so we can upgrade/install grub. Your paths may vary depending on your drive configuration.

• sudo mkdir /media/RESCUE
• sudo mount /dev/pve/root /media/RESCUE/
• sudo mount /dev/[partition of efi, for me nvme0n1p2] /media/RESCUE/boot/efi
• sudo mount -t proc proc /media/RESCUE/proc
• sudo mount -t sysfs sys /media/RESCUE/sys
• sudo mount -o bind /dev /media/RESCUE/dev
• sudo mount -o bind /run /media/RESCUE/run

Chroot into your proxmox install.

• chroot /media/RESCUE

Since my UUID changed on hardware I had to extend my root

• lvextend -L +1g /dev/pve/root
• resize2fs /dev/pve/root

You could also rename the vg and rename it back instead of the last step.

Then update grub and install it.

• update-grub
• grub-install

If there are no error messages, you should be able to reboot now.

here is the bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=987008

Please Post the logs.


Note down the name of the faulty disk and from the command line or the resuce interface with your favourite edit comment out the disk name from the /etc/fstab file with a #.

Try a reboot and that should help you boot to pve but without mounting the disk which has been commented out. after that use the pve shell to share logs and even try trouble shooting the issue.


Best Regards

 

[dudis]

Thank you for sharing!

It happened yesterday here too. :/

I follow your script and it works! Thanks again!

 

[openaspace]

Solved with this
https://pve.proxmox.com/wiki/Recover_From_Grub_Failure

 

[lucius_the]

Had the same thing today. Just a regular PVE update and after it -> grub can't find a disk.
I did notice, while apt upgrade was in progress, that grub-update prober couldn't find something... But I didn't think it was anything important finally solved by using a live ISO and lvresize, like others have pointed.