[SOLVED] "Welcome to GRUB" after apt dist-upgrade

P

pigpen

New Member
Dec 13, 2023
15
3
3
I did an update today:

Start-Date: 2025-02-28 07:25:33
Commandline: apt-get dist-upgrade
Upgrade: librados2:amd64 (17.2.7-pve3, 17.2.8-pve1), ceph-fuse:amd64 (17.2.7-pve3, 17.2.8-pve1), proxmox-widget-toolkit:amd64 (4.3.4, 4.3.6), libpve-rs-perl:amd64 (0.9.1, 0.9.2), libproxmox-acme-perl:amd64 (1.5.1, 1.6.0), python3-ceph-common:amd64 (17.2.7-pve3, 17.2.8-pve1), librbd1:amd64 (17.2.7-pve3, 17.2.8-pve1), grub-pc-bin:amd64 (2.06-13+pmx2, 2.06-13+pmx5), librgw2:amd64 (17.2.7-pve3, 17.2.8-pve1), ceph-common:amd64 (17.2.7-pve3, 17.2.8-pve1), libproxmox-rs-perl:amd64 (0.3.4, 0.3.5), grub-efi-amd64:amd64 (2.06-13+pmx2, 2.06-13+pmx5), python3-cephfs:amd64 (17.2.7-pve3, 17.2.8-pve1), libcephfs2:amd64 (17.2.7-pve3, 17.2.8-pve1), libproxmox-acme-plugins:amd64 (1.5.1, 1.6.0), pve-i18n:amd64 (3.3.3, 3.4.0), libradosstriper1:amd64 (17.2.7-pve3, 17.2.8-pve1), grub-efi-amd64-bin:amd64 (2.06-13+pmx2, 2.06-13+pmx5), grub2-common:amd64 (2.06-13+pmx2, 2.06-13+pmx5), python3-rbd:amd64 (17.2.7-pve3, 17.2.8-pve1), python3-rgw:amd64 (17.2.7-pve3, 17.2.8-pve1), grub-common:amd64 (2.06-13+pmx2, 2.06-13+pmx5), python3-ceph-argparse:amd64 (17.2.7-pve3, 17.2.8-pve1), python3-rados:amd64 (17.2.7-pve3, 17.2.8-pve1)
End-Date: 2025-02-28 07:25:54

So, these Grub packages were updated:

grub-pc-bin
grub-efi-amd64
grub-efi-amd64-bin
grub2-common
grub-common

After rebooting the box just showed "Welcome to GRUB" without the possibility to type anything.
So I went into the BIOS and saw that under the NVME there was an EFI entry named "proxmox" which I selected and it booted successfully again!

Then I thought: This is a good opportunity to flash the latest BIOS from the vendor (it's a Shuttle DL20N6).
After that, there was no more EFI entry named "proxmox", only the NVME itself.
The "Welcome to GRUB" message was back and Proxmox did not boot!

So I booted Ubuntu via Ventoy and chrooted myself into Proxmox ...

I did a lot of things like:
apt reinstall grub-efi-amd64
grub-install /dev/nvme0n1
...

All to no avail ...

After a bit of Googling I saw that the file /etc/kernel/proxmox-boot-uuids did not exist, so I tried:

umount /boot/efi
proxmox-boot-tool init /dev/nvme0n1p2

There still is no separate EFI entry named "proxmox", but just booting from the NVME now works again!
proxmox-boot-tool status now says:


root@prox:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
F1B8-EA75 is configured with: grub (versions: 6.5.13-6-pve, 6.8.12-7-pve, 6.8.12-8-pve)

So all seems fine again.

My question is: Is my Proxmox set up correctly? What things can I check in order to prevent this from happening again?

Grub packages that are currently installed:

root@prox:~# dpkg-query -l | grep grub
ii grub-common 2.06-13+pmx5 amd64 GRand Unified Bootloader (common files)
ii grub-efi-amd64 2.06-13+pmx5 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 version)
ii grub-efi-amd64-bin 2.06-13+pmx5 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 modules)
rc grub-pc 2.06-13+pmx2 amd64 GRand Unified Bootloader, version 2 (PC/BIOS version)
ii grub-pc-bin 2.06-13+pmx5 amd64 GRand Unified Bootloader, version 2 (PC/BIOS modules)
ii grub2-common 2.06-13+pmx5 amd64 GRand Unified Bootloader (common files for version 2)
 
Last edited:
  • Like
Reactions: Shack6875
Sir, I would like to thank you for your guide.
Due to a power loss and faulty UPS behavior, my server was this morning in the same state.
Now I'm finally able to boot as before.
 
  • Like
Reactions: pigpen
Hello,

I've got the same problem after the update, which I'd installed today.
Proxmox is installed on a Lenovo Tiny computer with a NVMe SSD installed.

After changing the BIOS boot settings from UEFI to Legacy, Proxmox started normally and everything is working fine now....

Rangald
 
  • Like
Reactions: pigpen
pigpen said:
So I booted Ubuntu via Ventoy and chrooted myself into Proxmox ...
Click to expand...
pigpen said:
After a bit of Googling I saw that the file /etc/kernel/proxmox-boot-uuids did not exist, so I tried:

umount /boot/efi
proxmox-boot-tool init /dev/nvme0n1p2
Click to expand...
I'm in the same situation right now.
Could you show the exact commands you used to "Chroot into Proxmox".

Thanks.
 
gfngfn256 said:
Could you show the exact commands you used to "Chroot into Proxmox".
Click to expand...
Replying to my own post:

What I did;

1. Booted up into Proxmox Rescue Mode (with the 8.3 ISO)
2. Logged in
3.
Code: 
umount /boot/efi
proxmox-boot-tool init /dev/nvme0n1p2

Boots up now.
 
  • Like
Reactions: pigpen
Just for the sake of anyone coming here with the same issue.

I believe since I always never had proxmox-boot-tool status uuids showing, (this I have always believed came from the installation/update history of this specific machine), upon last weeks update of:
Code: 
Get:14 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub2-common amd64 2.06-13+pmx5 [617 kB]
Get:15 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-efi-amd64 amd64 2.06-13+pmx5 [45.7 kB]
Get:16 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-efi-amd64-bin amd64 2.06-13+pmx5 [1577 kB]
Get:17 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-pc-bin amd64 2.06-13+pmx5 [1002 kB]
Get:18 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-common amd64 2.06-13+pmx5 [2717 kB]
The machine will not boot up again if restarted.

The fix (at least for me) is as in my above post.

Users who don't have a physical console/monitor connected to their server - are advised to check the proxmox-boot-tool status before attempting a reboot of the server - which as stated may fail! They could then (at their own responsibility) commit the above change before rebooting.
 
  • Like
Reactions: pigpen
Nice, I didn't even know there was a Proxmox Rescue Mode! Will try that next time if needed!
But the original question remains: Was there a bug in the updated packages or are our systems configured incorrectly?
 
I actually don't think there is/was a bug in those updates, but rather our servers/grub status have always been compromised with their grub/bootloaders.

On many updates I would see that the proxmox-boot-tool would indicate it does not have any uuids - and I would ignore this, with the working assumption; "If it ain't broke don't fix it", so I knew of the above proxmox-boot-tool init command & its documentation, studied it somewhat, but felt better to leave it alone.

Until last night - when we had a power-loss event & I had to reboot the server.....
 
Last edited:
If it's not a bug then there had to be a breaking change at least.
In the documentation you mentioned it reads:

Both update-initramfs and apt (when necessary) will automatically trigger a refresh.

So I hope we are now safe and incoming updates and reboots will just work as expected.
@staff: Is this a safe assumption?
 
  • Like
Reactions: gfngfn256
proxmox-boot-tool not showing any UUIDs is fine on systems that don't use proxmox-boot-tool to manage their ESPs (such as a regular install on top of LVM).

could you post the entry form /var/log/apt/term.log covering the upgrade? "efibootmgr -v" output and information with which version of the ISO your system was installed would also be interesting!

I suspect something was wrong with the initial bootloader setup, but hard to tell without more information..
 
I believe this is the update which caused my node to become unbootable:
Code: 
Starting system upgrade: apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  ceph-common ceph-fuse grub-common grub-efi-amd64 grub-efi-amd64-bin grub-pc-bin
  grub2-common libcephfs2 libproxmox-acme-perl libproxmox-acme-plugins
  libproxmox-rs-perl libpve-rs-perl librados2 libradosstriper1 librbd1 librgw2
  proxmox-widget-toolkit pve-i18n python3-ceph-argparse python3-ceph-common
  python3-cephfs python3-rados python3-rbd python3-rgw
24 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 50.1 MB of archives.
After this operation, 294 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 librgw2 amd64 17.2.8-pve1 [5598 kB]
Get:2 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 libradosstriper1 amd64 17.2.8-pve1 [440 kB]
Get:3 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 ceph-common amd64 17.2.8-pve1 [25.5 MB]
Get:4 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 librbd1 amd64 17.2.8-pve1 [3333 kB]
Get:5 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 librados2 amd64 17.2.8-pve1 [3432 kB]
Get:6 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 python3-cephfs amd64 17.2.8-pve1 [144 kB]
Get:7 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 libcephfs2 amd64 17.2.8-pve1 [699 kB]
Get:8 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 python3-rgw amd64 17.2.8-pve1 [92.0 kB]
Get:9 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 python3-rados amd64 17.2.8-pve1 [293 kB]
Get:10 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 python3-ceph-argparse all 17.2.8-pve1 [24.1 kB]
Get:11 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 python3-ceph-common all 17.2.8-pve1 [50.6 kB]
Get:12 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 python3-rbd amd64 17.2.8-pve1 [283 kB]
Get:13 http://download.proxmox.com/debian/ceph-quincy bookworm/no-subscription amd64 ceph-fuse amd64 17.2.8-pve1 [805 kB]
Get:14 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub2-common amd64 2.06-13+pmx5 [617 kB]
Get:15 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-efi-amd64 amd64 2.06-13+pmx5 [45.7 kB]
Get:16 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-efi-amd64-bin amd64 2.06-13+pmx5 [1577 kB]
Get:17 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-pc-bin amd64 2.06-13+pmx5 [1002 kB]
Get:18 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 grub-common amd64 2.06-13+pmx5 [2717 kB]
Get:19 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 libproxmox-acme-plugins all 1.6.0 [139 kB]
Get:20 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 libproxmox-acme-perl all 1.6.0 [11.3 kB]
Get:21 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 libproxmox-rs-perl all 0.3.5 [4180 B]
Get:22 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 libpve-rs-perl amd64 0.9.2 [2589 kB]
Get:23 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 proxmox-widget-toolkit all 4.3.6 [221 kB]
Get:24 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 pve-i18n all 3.4.0 [418 kB]
Fetched 50.1 MB in 50s (1009 kB/s)                                                 
Reading changelogs... Done
Preconfiguring packages ...
(Reading database ... 60419 files and directories currently installed.)
Preparing to unpack .../00-librgw2_17.2.8-pve1_amd64.deb ...
Unpacking librgw2 (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../01-libradosstriper1_17.2.8-pve1_amd64.deb ...
Unpacking libradosstriper1 (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../02-ceph-common_17.2.8-pve1_amd64.deb ...
Unpacking ceph-common (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../03-librbd1_17.2.8-pve1_amd64.deb ...
Unpacking librbd1 (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../04-librados2_17.2.8-pve1_amd64.deb ...
Unpacking librados2 (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../05-python3-cephfs_17.2.8-pve1_amd64.deb ...
Unpacking python3-cephfs (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../06-libcephfs2_17.2.8-pve1_amd64.deb ...
Unpacking libcephfs2 (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../07-python3-rgw_17.2.8-pve1_amd64.deb ...
Unpacking python3-rgw (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../08-python3-rados_17.2.8-pve1_amd64.deb ...
Unpacking python3-rados (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../09-python3-ceph-argparse_17.2.8-pve1_all.deb ...
Unpacking python3-ceph-argparse (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../10-python3-ceph-common_17.2.8-pve1_all.deb ...
Unpacking python3-ceph-common (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../11-python3-rbd_17.2.8-pve1_amd64.deb ...
Unpacking python3-rbd (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../12-ceph-fuse_17.2.8-pve1_amd64.deb ...
Unpacking ceph-fuse (17.2.8-pve1) over (17.2.7-pve3) ...
Preparing to unpack .../13-grub2-common_2.06-13+pmx5_amd64.deb ...
Unpacking grub2-common (2.06-13+pmx5) over (2.06-13+pmx2) ...
Preparing to unpack .../14-grub-efi-amd64_2.06-13+pmx5_amd64.deb ...
Unpacking grub-efi-amd64 (2.06-13+pmx5) over (2.06-13+pmx2) ...
Preparing to unpack .../15-grub-efi-amd64-bin_2.06-13+pmx5_amd64.deb ...
Unpacking grub-efi-amd64-bin (2.06-13+pmx5) over (2.06-13+pmx2) ...
Preparing to unpack .../16-grub-pc-bin_2.06-13+pmx5_amd64.deb ...
Unpacking grub-pc-bin (2.06-13+pmx5) over (2.06-13+pmx2) ...
Preparing to unpack .../17-grub-common_2.06-13+pmx5_amd64.deb ...
Unpacking grub-common (2.06-13+pmx5) over (2.06-13+pmx2) ...
Preparing to unpack .../18-libproxmox-acme-plugins_1.6.0_all.deb ...
Unpacking libproxmox-acme-plugins (1.6.0) over (1.5.1) ...
Preparing to unpack .../19-libproxmox-acme-perl_1.6.0_all.deb ...
Unpacking libproxmox-acme-perl (1.6.0) over (1.5.1) ...
Preparing to unpack .../20-libproxmox-rs-perl_0.3.5_all.deb ...
Unpacking libproxmox-rs-perl (0.3.5) over (0.3.4) ...
Preparing to unpack .../21-libpve-rs-perl_0.9.2_amd64.deb ...
Unpacking libpve-rs-perl (0.9.2) over (0.9.1) ...
Preparing to unpack .../22-proxmox-widget-toolkit_4.3.6_all.deb ...
Unpacking proxmox-widget-toolkit (4.3.6) over (4.3.4) ...
Preparing to unpack .../23-pve-i18n_3.4.0_all.deb ...
Unpacking pve-i18n (3.4.0) over (3.3.3) ...
Setting up librados2 (17.2.8-pve1) ...
Setting up proxmox-widget-toolkit (4.3.6) ...
Setting up librgw2 (17.2.8-pve1) ...
Setting up python3-ceph-argparse (17.2.8-pve1) ...
Setting up libproxmox-acme-perl (1.6.0) ...
Setting up libcephfs2 (17.2.8-pve1) ...
Setting up grub-common (2.06-13+pmx5) ...
Setting up libradosstriper1 (17.2.8-pve1) ...
Setting up libproxmox-acme-plugins (1.6.0) ...
Setting up python3-ceph-common (17.2.8-pve1) ...
Setting up pve-i18n (3.4.0) ...
Setting up librbd1 (17.2.8-pve1) ...
Setting up grub-efi-amd64-bin (2.06-13+pmx5) ...
Setting up ceph-fuse (17.2.8-pve1) ...
Setting up python3-rados (17.2.8-pve1) ...
Setting up grub2-common (2.06-13+pmx5) ...
Setting up python3-rbd (17.2.8-pve1) ...
Setting up grub-pc-bin (2.06-13+pmx5) ...
Setting up grub-efi-amd64 (2.06-13+pmx5) ...
Installing for x86_64-efi platform.
File descriptor 3 (pipe:[39572520]) leaked on vgs invocation. Parent PID 3266674: grub-install.real
File descriptor 3 (pipe:[39572520]) leaked on vgs invocation. Parent PID 3266674: grub-install.real
Installation finished. No error reported.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.8.12-8-pve
Found initrd image: /boot/initrd.img-6.8.12-8-pve
Found linux image: /boot/vmlinuz-6.8.12-7-pve
Found initrd image: /boot/initrd.img-6.8.12-7-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Adding boot menu entry for UEFI Firmware Settings ...
done
Setting up python3-rgw (17.2.8-pve1) ...
Setting up python3-cephfs (17.2.8-pve1) ...
Setting up ceph-common (17.2.8-pve1) ...
Setting system user ceph properties..usermod: no changes
usermod: unlocking the user's password would result in a passwordless account.
You should set a password with usermod -p to unlock this user's password.
..done
chown: cannot access '/var/log/ceph/*.log*': No such file or directory
Fixing /var/run/ceph ownership....done
Setting up libproxmox-rs-perl (0.3.5) ...
Setting up libpve-rs-perl (0.9.2) ...
Processing triggers for pve-ha-manager (4.0.6) ...
Processing triggers for libc-bin (2.36-9+deb12u9) ...
Processing triggers for pve-manager (8.3.4) ...
Processing triggers for man-db (2.11.2-2) ...
Removing subscription nag from UI...

Your System is up-to-date

starting shell
root@

See this older post of mine, which may also shed some light on the situation.

I've already posted above the resolution that worked for me.
 
just to double check - there was no error message shown on the screen whatsoever, just the boot didn't proceed? could you still post your current "efibootmgr -v" output?
 
Blank screen with just "Welcome to GRUB" in reversed black-on-white text, something like this:Untitled.png

fabian said:
current "efibootmgr -v" output
Click to expand...
Code: 
efibootmgr -v
BootCurrent: 0001
Timeout: 1 seconds
BootOrder: 0001,0000
Boot0000* proxmox       HD(2,GPT,9cbeef3c-23f6-4dec-89ca-dc71a7bf1903,0x800,0x200000)/File(\EFI\proxmox\grubx64.efi)
Boot0001* UEFI OS       HD(2,GPT,9cbeef3c-23f6-4dec-89ca-dc71a7bf1903,0x800,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)..BO
 
okay - so your boot order is currently wrong, you are booting the "fallback" bootloader that gets installed for compatibility with broken EFI implementations.. does your system boot if you select the "proxmox" boot entry (then "BootCurrent" should say "0000")?
 
fabian said:
does your system boot if you select the "proxmox" boot entry (then "BootCurrent" should say "0000")?
Click to expand...
Presently I can't tell you that - since I have no physical monitor attached, but previously (pre-resolution, see above) when I had the above problem & I had the device connected to a monitor - I entered the BIOS settings, messed around a little ... found the "proxmox" bootloader entry, but did not manage to make it boot. Later on that "proxmox" entry/option disappeared. I may be at fault there - as I started attempting the various boot options.

Do you want me to attempt to change the boot order with efibootmgr -o 0,1 & then reboot?
I may have to attach a monitor in the end to troubleshoot, if things go wrong, but I'm willing to help.
 
assuming that is the ESP that you freshly initialized, both should be identical. you can verify by comparing the checksums ;)
 
Code: 
~# md5sum  /boot/efi/EFI/proxmox/grubx64.efi
74308e450e8552649176dc5f60be39cb  /boot/efi/EFI/proxmox/grubx64.efi
~# md5sum /boot/efi/EFI/BOOT/BOOTx64.EFI
74308e450e8552649176dc5f60be39cb  /boot/efi/EFI/BOOT/BOOTx64.EFI

Looks OK.
 
yeah, so switching the bootorder *should* work unless your EFI implementation is somehow really broken..
 
fabian said:
yeah, so switching the bootorder *should* work unless your EFI implementation is somehow really broken..
Click to expand...
I guess even if it fails to boot, I should be able to boot with USB Proxmox ISO, Rescue Mode & then switch back the boot order?
 
You must log in or register to reply here.
Top Bottom