Proxmox update from 7.2-4 GRUB update failure

[kocherjj]

I tried to upgrade a proxmox server from 7.2-4 today. The upgrade process seemed to work fine until the end when it failed on a GRUB error.


When I select 'No' I see this momentarily...


...and then this.


When I press 'Ok' I get this list of options.


Pressing 'Ok" here just takes me back through the loop.

I have both browser and SSH access to the server, and here is the partition layout in the browser interface.


It looks to me as if GRUB should be installing to /dev/sda1, not /dev/sda3, but this option is not showing up in the install menu. Any idea what would cause this error and how I can repair it without taking the server down? I have upgraded the server numerous times without issue. At the moment I can leave it like this since it is on a UPS but obviously this can't remain in this state of limbo forever.

Thank you,
Judah

 

[Neobin]

Only wild guessing:

Today/Yesterday Debian 11.5 got released [1] and with it an update for grub2.
Maybe there are some adjustments/changes needed on the Proxmox-side?
Possibly the Proxmox-Staff (e.g.: @t.lamprecht ) can/want to shed some light on this after the weekend.

[1] https://www.debian.org/News/2022/2022091002.en.html

 

[nontijt]

I have exactly the same situation. I posted yesterday this as well. But my thread needs to be approved first
ONly difference is that I run (boot) proxmox on a zfs mirror

I have left it waiting in the "Configure GRUB-PC" screen for now. waiting advice as well

[edit]
my thread has been made public.
For reference to this grub 'issue', here is the link
https://forum.proxmox.com/threads/update-proxmox-results-in-grub-question.114945/

 

[leesteken]

It's strange that /dev/sda3 got selected. I expect that deselecting that one (and selecting only /dev/sda) fixes the issue (for post #1).

 

[janssensm]

I would assume that booting with proxmox-boot-tool in combination with legacy bios, that no devices need to be selected as proxmox-boot-tool would take control.

But then you have to be very sure that you are using it. Wise to wait a bit until one of the staff members responds.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_footnote_3
So installed from pve 6.4 or later.

Or in case of an upgraded older install before pve 6.4 and you did the switch from legacy to proxmox-boot-tool yourself manually:
https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool

 

[WHSProxmox]

Same here ... also waiting in Update dialog. My / was on
root@prox3:~# findmnt / TARGET SOURCE FSTYPE OPTIONS / /dev/mapper/pve-root ext4 rw,relatime,errors=remount-ro

The boot disk with pve on it is sda ...

Quick suggestion how to proceed are welcome!

 

[WHSProxmox]

Same here ... also waiting in Update dialog. My / was on
root@prox3:~# findmnt / TARGET SOURCE FSTYPE OPTIONS / /dev/mapper/pve-root ext4 rw,relatime,errors=remount-ro

The boot disk with pve on it is sda ...

Quick suggestion how to proceed are welcome!

I go into risk check per dd that grub was instelled on sda and only select this disk in the dialog.

Reboot went well, uff.

 

[Darkhand]

I'm having a similar issue, this grub upgrade does need looking at on the Proxmox side:

 

[uncle.cripple]

Same problems with the Update on two Proxmox Servers, running with ZFS-mirror Boot devices.

proxmox-ve: 7.2-1 (running kernel: 5.15.53-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-10
pve-kernel-helper: 7.2-10
pve-kernel-5.13: 7.1-9
pve-kernel-5.4: 6.4-10
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.15.35-1-pve: 5.15.35-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-5-pve: 5.13.19-13
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.4.151-1-pve: 5.4.151-1
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-3
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1

 

[ednt]

Same problem with sda and sdb
During the install loop I can see:

 

[cc_mha]

same here - zfs mirror as boot device.

# proxmox-boot-tool status Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace.. System currently booted with legacy bios BF9D-115B is configured with: uefi (versions: 5.0.21-5-pve, 5.4.106-1-pve, 5.4.78-2-pve), grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-4-pve) BF9D-4BE5 is configured with: uefi (versions: 5.0.21-5-pve, 5.4.106-1-pve, 5.4.78-2-pve), grub (versions: 5.13.19-6-pve, 5.15.39-1-pve, 5.15.39-4-pve)

 

[Neobin]

Edit: Seems to be a general problem; not only in combination with Proxmox. [1] [2]

[1] https://forum.proxmox.com/threads/r...boot-uuids-does-not-exist.114998/#post-497164
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=#1019564

 

[Stoiko Ivanov]

Thanks for the report to everyone!

I think the issue in this case is only happening on proxmox-systems - and we have an initial patch on the mailing list possibly addressing it:
https://lists.proxmox.com/pipermail/pve-devel/2022-September/053894.html

You can manually apply the patch and rerun the upgrade - or can alternatively run grub-install.real <all-devices-you-have-grub-installed-on>

 

[nontijt]

Thanks for the report to everyone!

I think the issue in this case is only happening on proxmox-systems - and we have an initial patch on the mailing list possibly addressing it:
https://lists.proxmox.com/pipermail/pve-devel/2022-September/053894.html

You can manually apply the patch and rerun the upgrade - or can alternatively run grub-install.real <all-devices-you-have-grub-installed-on>

Thanks for this good news!

For the people that are currently in the update process, and see the selection screen for drives, what is the recommend course? What to select.

I am still waiting in "GRUB install devices" screen.

 

[Stoiko Ivanov]

A version of pve-kernel-helper containing the fix is now available on pvetest.
You can install it from there - and depending on whether your system uses proxmox-boot-tool or not run `proxmox-boot-tool reinit` (if it does) or grub-install.real </path/to/grub/device>

We would be grateful for feedback if the fix indeed works as expected for you!

 

[nontijt]

A version of pve-kernel-helper containing the fix is now available on pvetest.
You can install it from there - and depending on whether your system uses proxmox-boot-tool or not run `proxmox-boot-tool reinit` (if it does) or grub-install.real </path/to/grub/device>

We would be grateful for feedback if the fix indeed works as expected for you!

Ok, great. But at this moment, i am this in the middle of the upgrade. I just did not continue the selection.

I have read some others just stopped there waiting on advice.

What to do in that situation?
Close the setup screen? Select x.. or y...
Thanks in advance.

 

[genguser]

Ok, great. But at this moment, i am this in the middle of the upgrade. I just did not continue the selection.

I have read some others just stopped there waiting on advice.

What to do in that situation?
Close the setup screen? Select x.. or y...
Thanks in advance.

Hi nontijt. I'm in same situation as you and many more people. Except that i did already chose yes for grub during upgrade. So I'am stuck with non bootable bios grub system now.
I will try that new pve-kernel-helper from pvetest repo. Next i'll inform here if it helps. Good luck for you.
My hardware is Proliant DL380e G8.

 

[Stoiko Ivanov]

What to do in that situation?
Close the setup screen? Select x.. or y...
Thanks in advance.

In that situation the grub-install will not be successful, unless you manually patch the file /usr/sbin/grub-install with the patch right now in a different shell
so - unless you want to do that - select to continue with the failing grub-install and run the commands from above after the upgrade

I hope this helps!

 

[genguser]

I can confirm that after updating pve-kernel-helper to version 7.2-12 from pvetest repo and entering command grub-install.real </path/to/grub/device> which mentioned Mr. Stoiko Ivanov is everything running ok. Clear boot with everything running.

 

[nontijt]

In that situation the grub-install will not be successful, unless you manually patch the file /usr/sbin/grub-install with the patch right now in a different shell
so - unless you want to do that - select to continue with the failing grub-install and run the commands from above after the upgrade

I hope this helps!

can you explain to me how to do that patching the file /usr/sbin/grub-install ?

If it is easy, I will do it directly