[SOLVED] PVE 9 - GRUB stops the machine immediately

[fabian]

so you need to add an entry there for your ESP (/dev/nvme0n1p2) to mount it at /boot/efi .. and then you should reinstall your bootloader (apt install --reinstall grub-efi-amd64)

 

[Privacy]

Thank you!
But that sounds a bit different compared to the information on the help page:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_proxmox_boot_tool

Can you please explain that a bit more?

To setup an existing, unmounted ESP located on /dev/sda2 for inclusion in Proxmox VE’s kernel update synchronization mechanism, use the following:
# proxmox-boot-tool init /dev/sda2

So couldn't I just use # proxmox-boot-tool init /dev/nvme0n1p2 to setup my existing unmounted ESP?

I added

/dev/nvme0n1p2 /boot/efi vfat defaults 0 1

to /etc/fstab.

 

[Privacy]

So couldn't I just use # proxmox-boot-tool init /dev/nvme0n1p2 to setup my existing unmounted ESP?

I'll answer myself: yes, you can.

I did it, rebooted and it worked.

 

[fabian]

sure, you can also use proxmox-boot-tool to manage the ESP, but for your setup there is no advantage..

 

[Privacy]

sure, you can also use proxmox-boot-tool to manage the ESP, but for your setup there is no advantage..

I see. What would be an advantage over a system which is able to boot with no issues again?

 

[fabian]

with proxmox-boot-tool, all kernels and initrds are copied to the ESP - which means you are limited by the amount of space on the ESP, and can't keep all kernel versions bootable if you have many of them.

 

[Privacy]

Thanks a lot for the explanation.
Let's see if I can remember that in case I'll get space issues in future.
But for now I think I'm fine:

Filesystem             Size  Used Avail Use% Mounted on
/dev/nvme0n1p2        1022M  289M  734M  29% /boot/efi

An extra "Many thanks" for your ongoing assistance.

 

[thatgermandude]

Hey everyone, just wanted to chip in since I had the exact same Problem. Upgrade from 8 to 9 according to the official docs went smooth but then after a reboot I got stuck in a bootloop. It would just show "Welcome to grub" and then reboot.

Here is how I solved it:

I downloaded and booted from the official proxmox 9 iso and chose the rescue mode under advanced options.
After the System booted I could login through ssh. After reading the docs I tried to mount the efi partition first to check if the partition itself was ok:

root@pve:/# mount /dev/nvme0n1p2 /boot/efi
root@pve:/#
root@pve:/#
root@pve:/# ls -la /boot/efi/
total 12
drwxr-xr-x 3 root root 4096 Jan  1  1970 .
drwxr-xr-x 5 root root 4096 Sep 14 13:34 ..
drwxr-xr-x 4 root root 4096 Mar  8  2023 EFI

Then I just ran the grub commands:

root@pve:/# grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox /dev/nvme0n1
Installing for x86_64-efi platform.
Installation finished. No error reported.
root@pve:/#
root@pve:/# grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox --removable /dev/nvme0n1
Installing for x86_64-efi platform.
Installation finished. No error reported.
root@pve:/#
root@pve:/# update-initramfs -u -k all

The initramfs command would actually give me a few warnings about the removable bootloader and that grub would not update it.
But after a reboot everything was just working fine again.

Thanks for all your help

 

[glockmane]

In my case I did the following:

1) Boot with Proxmox ISO (Recovery Boot)
2) Shell:

mount /dev/sdX2 /boot/efi
grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox /dev/sdX
grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox --removable /dev/sdX
echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u
apt install --reinstall grub-efi-amd64
update-initramfs -u -k all

Now boot works again!

Can somebody explain what the "--removable" option is for and why this line is needed?

Also can somebody explain the echo line? Proxmox suggested me to do that and reinstall grub afterwards..

 

[Labomba]

Hey everyone, just wanted to chip in since I had the exact same Problem. Upgrade from 8 to 9 according to the official docs went smooth but then after a reboot I got stuck in a bootloop. It would just show "Welcome to grub" and then reboot.

Here is how I solved it:

I downloaded and booted from the official proxmox 9 iso and chose the rescue mode under advanced options.
After the System booted I could login through ssh. After reading the docs I tried to mount the efi partition first to check if the partition itself was ok:

root@pve:/# mount /dev/nvme0n1p2 /boot/efi
root@pve:/#
root@pve:/#
root@pve:/# ls -la /boot/efi/
total 12
drwxr-xr-x 3 root root 4096 Jan  1  1970 .
drwxr-xr-x 5 root root 4096 Sep 14 13:34 ..
drwxr-xr-x 4 root root 4096 Mar  8  2023 EFI

Then I just ran the grub commands:

root@pve:/# grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox /dev/nvme0n1
Installing for x86_64-efi platform.
Installation finished. No error reported.
root@pve:/#
root@pve:/# grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox --removable /dev/nvme0n1
Installing for x86_64-efi platform.
Installation finished. No error reported.
root@pve:/#
root@pve:/# update-initramfs -u -k all

The initramfs command would actually give me a few warnings about the removable bootloader and that grub would not update it.
But after a reboot everything was just working fine again.

Thanks for all your help

Thanks man, did the same and it worked!

 

[ewsclass66]

Can confirm this also fixed the same issue for me.

 

[Zolt]

Thanks for the tips thatgermandude and glockmane. I hit the same issue last night. What was frustrating is that the server cleared all pre-checks before I did the upgrade.

I have another two PVE servers that have failures on the pre-checks, hence the reason I did the other one first

 

[Zolt]

Second node upgrade shit the bed as well.

This node had a fail warning before the upgrade:

FAIL: systemd-boot meta-package installed. This will cause problems on upgrades of other boot-related packages.

So I made certain that I fixed that and re-ran the upgrade check before doing the upgrade.

After the upgrade it got stuck in the same old welcome to grub reboot loop. I thought no worries, I had the PVE 9.1 USB stick good to go. But things got even more fucking weird, as the PVE installer menu would appear and the USB keyboard would stop responding, so I couldn't select the rescue option.

Thought it was the keyboard, so tried a different one (I should note the keyboards were working fine to select BIOS options, and bring up the BIOS boot menu). Again same result with different keyboard and different USB ports, keyboard would wouldn't let me select the options in the PVE installer menu.

Thought maybe the installer was buggered, so tried another USB stick and re-imaged it. Same result.

Desperate times, so I went into the storage under my house to dig out a PS/2 to USB keyboard adapter... (thinking surely this won't work)... well it worked, was able to use it to finally go through to the PVE installer menus. No idea if the PVE installer does something funky with the USB ports while it is running up?

I might sell tickets to the upgrade of my third node.

I'm wondering if these issues are impacting users who have upgraded from PVE 7 to 8 to 9 over the years. I am guessing that upgrade path doesn't work at all. Only users who go from a clean PVE 8 to 9.

 

[fabian]

Second node upgrade shit the bed as well.

This node had a fail warning before the upgrade:



So I made certain that I fixed that and re-ran the upgrade check before doing the upgrade.

After the upgrade it got stuck in the same old welcome to grub reboot loop. I thought no worries, I had the PVE 9.1 USB stick good to go. But things got even more fucking weird, as the PVE installer menu would appear and the USB keyboard would stop responding, so I couldn't select the rescue option.

Thought it was the keyboard, so tried a different one (I should note the keyboards were working fine to select BIOS options, and bring up the BIOS boot menu). Again same result with different keyboard and different USB ports, keyboard would wouldn't let me select the options in the PVE installer menu.

Thought maybe the installer was buggered, so tried another USB stick and re-imaged it. Same result.

Desperate times, so I went into the storage under my house to dig out a PS/2 to USB keyboard adapter... (thinking surely this won't work)... well it worked, was able to use it to finally go through to the PVE installer menus. No idea if the PVE installer does something funky with the USB ports while it is running up?

I might sell tickets to the upgrade of my third node.

I'm wondering if these issues are impacting users who have upgraded from PVE 7 to 8 to 9 over the years. I am guessing that upgrade path doesn't work at all. Only users who go from a clean PVE 8 to 9.

could you post the output of pve8to9 as well as "proxmox-boot-tool status" and "efibootmgr -v" for the third node, *before doing the upgrade*? thanks!

 

[glockmane]

I upgraded about 10 PVE Hosts now with different problems, here's my notes, maybe it helps:

## Update PVE

apt update
apt dist-upgrade
reboot

##

## Upgrade Check

pve8to9 --full

##

## Fix Warnings/Errors

Optional: Remove systemd-boot

apt remove systemd-boot

Optional: Fix Removable Bootloader

echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u
apt install --reinstall grub-efi-amd64

Optional: Add non-free-firmware repository

nano /etc/apt/sources.list
deb http://ftp.de.debian.org/debian bookworm main contrib non-free-firmware

##

## Correct Sources

sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
rm /etc/apt/sources.list.d/pve-enterprise.list
rm /etc/apt/sources.list.d/pve-install-repo.list
rm /etc/apt/sources.list.d/ceph.list
rm /etc/apt/sources.list.d/pvetest-for-beta.list

##

## When EFI Boot:

#sdX = boot drive
#sdX2 = EFI partition

mount /dev/sdX2 /boot/efi
grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox /dev/sdX
grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox --removable /dev/sdX
(echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u)
(apt install --reinstall grub-efi-amd64)
update-initramfs -u -k all

##

## Upgrade

apt update
apt dist-upgrade

## Check again

pve8to9 --full

## Cleanup

apt modernize-sources
rm /etc/apt/sources.list.d/pve-enterprise.sources

## Finish

reboot