Upgrade PVE 6.x to 7.x : Grub issues

I had exactly the same issue and I've been busy the whole day to fix this.
The solution was as follow:
- Execute all the steps from here: https://pve.proxmox.com/wiki/Recover_From_Grub_Failure
- the update-grub will fail. At this point you need to lvextend your logical volume with 2G for example.
lvextend -L 298G /dev/pve/root
resize2fs /dev/pve/root 298G
- Now you can try again the update-grub. And magically it doesn't give any error
- now you can finally run the grub-install. In my case it was grub-install /dev/nvme1n1 (my second ssd where proxmox is installed. I have a first ssd with Windows installed)

Thanks to Miquel and his post here https://forum.proxmox.com/threads/system-unbootable-grub-error-disk-lvmid-not-found.98761/ for the solution. I'm attaching a screen shot where we can see the grub-update before and after the lvextend. After this I ran grub-install /dev/nvme1n1 and reboot and promox was back.

View attachment 31296

Please forgive my ignorance, but I've run into the same problem and am working my way through the instructions in this thread. When I get to
  • sudo mount /dev/sda1 /media/RESCUE/boot in the linked support document, am I right in thinking I'd need to change this as set out below?
My boot drive is an NVME SSD whereas /dev/sda1 is just a backup storage drive. I have attached the output of lsblk. Should I modify the line above to read sudo mount /nvme0n1p1 /media/RESCUE/boot or would I need a different partition on that drive?

Thank you
 

Attachments

  • Screenshot 2022-09-18 at 21.31.51.png
    Screenshot 2022-09-18 at 21.31.51.png
    154 KB · Views: 39
Please forgive my ignorance, but I've run into the same problem and am working my way through the instructions in this thread. When I get to
  • sudo mount /dev/sda1 /media/RESCUE/boot in the linked support document, am I right in thinking I'd need to change this as set out below?
My boot drive is an NVME SSD whereas /dev/sda1 is just a backup storage drive. I have attached the output of lsblk. Should I modify the line above to read sudo mount /nvme0n1p1 /media/RESCUE/boot or would I need a different partition on that drive?

Thank you

Hello chrisg991.

I hope you will have solved your problem by now, but as I encountered exactly the same issue this morning, and my configuration is similar to yours (NVME SSD), I'm taking the time to answer your post, so that it can also help all those who would be in the same situation.

So yes, in our case we have to modify the following line:

sudo mount /dev/sda1 /media/RESCUE/boot

And we should replace it with :

sudo mount /dev/nvme0n1p3 /media/RESCUE/boot

Because the grub files are into /boot/grub on the root LVM partition (which is /dev/nvme0n1p3 in our case).

However, this certainly won't be enough, because you'll also have to mount the EFI partition into /media/RESCUE/boot/grub/efi, just like this :

sudo mount /dev/nvme0n1p2 /media/RESCUE/boot/grub/efi

Then you can go on with the instructions.

I my case, the command below didn't work :

Bash:
lvextend -L 298G /dev/pve/root

So I had to use this one instead :

Bash:
lvextend -l +100%FREE /dev/pve/root

Then I didn't have to resize2fs, and I could immediately run update-grub without error.

Finaly, I also had to change the last command which, in our case, should be :

Bash:
grub-install /dev/nvme0n1

Or, in case it doesn't work, this one :

Bash:
grub-install /dev/nvme0n1p3

I don't remember which one is right, but you can safely try the first grub-install command above, as it will just throw an error message if it's wrong, then try the second one.
 
Hey @Cedrox

Thanks for providing clarification on this as I myself ran into the same confusion against what specific mount points I should be using. However I've now ran into another issue trying to follow your appended suggestions to the procedure.

Based on your above comment, the partition that has the nested pve partitions/images is the one that should be mounted to /media/RESCUE/boot. When attempting this I get the following error:

mount: /media/RESCUE/boot: unknown filesystem type 'LVM2_member'

Any thoughts?
 

Attachments

  • 20230608_195704.jpg
    20230608_195704.jpg
    995.4 KB · Views: 10
Okay have an update to my issue as I figured out the solution... well a user /u/hairy_tick over on reddit figured it out after I posted to /r/proxmox. Given EFI on my dell r610, it does not have the requirement to mount /media/RESCUE/boot, given the boot directory is already populated when /dev/pve/root is mounted.

As mentioned above in Cedrox's last post, EFI will need to be added as a mount point. So for my environment, the instructions from the wiki post provided in em3034's previous post get changed to something like as follows:

  • sudo vgscan
  • sudo vgchange -ay
  • sudo mkdir /media/RESCUE
  • sudo mount /dev/pve/root /media/RESCUE/
  • mount /dev/sda2 /media/RESCUE/boot/efi
  • sudo mount -t proc proc /media/RESCUE/proc
  • sudo mount -t sysfs sys /media/RESCUE/sys
  • sudo mount -o bind /dev /media/RESCUE/dev
  • sudo mount -o bind /run /media/RESCUE/run
For the reference point for the efi boot, I believe that was documented well in Cedrox's last post above, as when running lsblk it should be the #2 partition under the boot drive, likely notated with 512M.

And then the rest of the aforementioned procedure from e3034's previous post should complete without issue.

The specific post on reddit where /u/hairy_tick provided solution can be found here. I'm also copying the specific response under that post here, incase it somehow gets deleted/lost.

#####TEXT BELOW IS FROM /u/hairy_tick COMMENT FROM REDDIT THREAD####

OK, I think I see now. Looking at my proxmox I don't have a separate partition for /boot, it is just part of the root filesystem. So you probably don't need to mount anything at /boot.

I believe the separate /boot partition is done on machines that boot with BIOS because limitations in the way BIOS was designed mean it can have trouble accessing anything but the front part of larger disks. So a separate partition ensures that those files are always at the front of the disk.

EFI doesn't have that problem with big disks, so no need for a separate /boot. But EFI relies on a special EFI partition so EFI can find the boot loader (grub). Based on the sizes of your partitions matching what I have on my system, I think that partition for you is sda2.

Now, to confirm that's all true for you. First, do "ls /media/RESCUE/boot". If it shows files there then the boot files are available and there isn't another thing to mount to get /boot. Then we can mount the suspected efi partition

mount /dev/sda2 /media/RESCUE/boot/efi

And to see if it looks right

ls /media/RESCUE/boot/efi

There should be a directory there called EFI. If that's all right so far, then you are almost done. Just need to

chroot /media/RESCUE
update-grub
grub-install /dev/sda
exit
umount /media/RESCUE/boot/efi
umount /media/RESCUE
reboot

And in case you want to know why those commands, here's a bit of an explanation for why you are doing those commands. Feel free to skip this, but I know I hate doing things without at least a slight idea of what they are doing and why I am running them.

chroot creates a restriction for its children where some directory will appear to them to be the root (/) directory, and starts a new shell there. The basic idea is to restrict a program so it can't mess with files that don't belong to it. This is the filesystem part of how containers work, so what is a directory on the host appears to be / to programs in the container. Chroot doesn't set up any of the other restrictions that containers get (which is good for what we are doing here).

Update-grub. We want to make sure grub's config files match the current state of the system. If it is for example looking for a kernel that doesn't exist anymore, it can cause booting problems like you saw. So this will regenerate that config based on what it finds in the system now. Normally this is run automatically when needed (for example, when a new kernel version is installed), but if for some reason that didn't happen it could cause exactly the sort of problems you are seeing. This won't see the files on the disk you booted from, but instead the files on the proxmox installation because of the chroot.

Grub-install /dev/sda - this one will re-install grub in the EFI. That's only sometimes needed because occasionally you get a problem where the grub config is fine, but grub can't actually find it. This would fix that if that's what you are seeing, and does no harm if that isn't the problem.

exit - leave the chroot-ed shell.

umount - unmount those things we mounted to finish any writes to them that aren't complete yet to prepare them for the reboot.

reboot - yeah, it does what it says it will.
 
Last edited:
  • Like
Reactions: Cedrox

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!