Update to 8.2 broke the entire system

[MrnMrk]

Hello, I really need some help:
Last night i launched an update from 8.1.3 to 8.2 using proxmox GUI. The update finished and it was prompt to reboot the system due to kernel change.
The system never booted ever again, i plugged a screen and i can see the boot being stuck at Loading linux <Kernel Version>-pve, and it remains stuck there.
Grub seems not to be working, I was following this guide https://pve.proxmox.com/wiki/Recover_From_Grub_Failure but i have no idea what partition is the boot one refered as "sda1".
Any help is very welcome.

 

[fiona]

Hi,

Hello, I really need some help:
Last night i launched an update from 8.1.3 to 8.2 using proxmox GUI. The update finished and it was prompt to reboot the system due to kernel change.
The system never booted ever again, i plugged a screen and i can see the boot being stuck at Loading linux <Kernel Version>-pve, and it remains stuck there.

That sounds like it's hanging after GRUB.

Grub seems not to be working, I was following this guide https://pve.proxmox.com/wiki/Recover_From_Grub_Failure but i have no idea what partition is the boot one refered as "sda1".
Any help is very welcome.

Can you boot into an older kernel? Please also see:
https://pve.proxmox.com/wiki/Roadmap#8.2-known-issues
In particular, you could try setting the intel_iommu=off option.

 

[MrnMrk]

Hi,

That sounds like it's hanging after GRUB.

Can you boot into an older kernel? Please also see:
https://pve.proxmox.com/wiki/Roadmap#8.2-known-issues
In particular, you could try setting the intel_iommu=off option.

Hi Fiona, thanks for stopping by. Want to mention that my system is AM4 AMD, i have IOMMU enabled for a GPU usage in a VM, how do i boot an older kernel?

 

[fiona]

Hi Fiona, thanks for stopping by. Want to mention that my system is AM4 AMD, i have IOMMU enabled for a GPU usage in a VM, how do i boot an older kernel?

Okay, than the Intel option does not matter for you. During early boot, you should be able to select the kernel (by selecting the Advanced options for Proxmox VE GNU/Linux).

 

[MrnMrk]

Okay, than the Intel option does not matter for you. During early boot, you should be able to select the kernel (by selecting the Advanced options for Proxmox VE GNU/Linux).

I dont get that option, its one of the problems i tried to face last night. The system Power on into motherboard logo then straight to Booting PVE. I also recorded a short clip but its too large to send here
https://streamable.com/kot1qq

 

[Der Harry]

Hi Fiona, thanks for stopping by. Want to mention that my system is AM4 AMD, i have IOMMU enabled for a GPU usage in a VM, how do i boot an older kernel?

Welcome to the club. It's Xeon, Intel, Epyc, ...

I have a solution - the (semi-)official Proxmox GmbH way is "something with your bios, e.g. update" or "pin the kernel".

root@proxmox:~# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
6.5.13-5-pve <<<<<<<<<<<<<<<< pick a 6.5x Kernel
6.8.4-2-pve
6.8.4-3-pve

Pinned kernel:
6.5.13-5-pve

# pin the version (you might have installed a different 6.5.x)
root@proxmox:~# proxmox-boot-tool kernel pin 6.5.13-5-pve

 

[MrnMrk]

Welcome to the club. It's Xeon, Intel, Epyc, ...

I have a solution - the (semi-)official Proxmox GmbH way is "something with your bios, e.g. update" or "pin the kernel".

root@proxmox:~# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
6.5.13-5-pve <<<<<<<<<<<<<<<< pick a 6.5x Kernel
6.8.4-2-pve
6.8.4-3-pve

Pinned kernel:
6.5.13-5-pve

# pin the version (you might have installed a different 6.5.x)
root@proxmox:~# proxmox-boot-tool kernel pin 6.5.13-5-pve

Right but i dont have the shell at all. As i was following the Grub failure guide, i am suppose to mount root and some other directories and then i can chroot, but im stuck at the dev/sda1 which i dont know what it is, boot partition.

 

[Der Harry]

Right but i dont have the shell at all. As i was following the Grub failure guide, i am suppose to mount root and some other directories and then i can chroot, but im stuck at the dev/sda1 which i dont know what it is, boot partition.

Well Imagine all the nice admins with a Hetzner Server

You can use stock debian 12 (no zfs) or any older proxmox e.g. 8.0/8.1 with a 6.2 / 6.5 to boot to get to a rescue shell.

... or any other rescue linux of course.

 

[MrnMrk]

Well Imagine all the nice admins with a Hetzner Server

You can use stock debian 12 (no zfs) or any older proxmox e.g. 8.0/8.1 with a 6.2 / 6.5 to boot to get to a rescue shell.

... or any other rescue linux of course.

Maybe I didnt explain myself correctly, I have booted 8.2 proxmox iso, launched the install and then CTRL + ALT + F3 gets you to the shell, and there i am

 

[fiona]

I dont get that option, its one of the problems i tried to face last night. The system Power on into motherboard logo then straight to Booting PVE. I also recorded a short clip but its too large to send here
https://streamable.com/kot1qq

Did you ever modify the timeout/menu settings in /boot/grub/grub.cfg? It should contain an entry

submenu 'Advanced options for Proxmox VE GNU/Linux' ...

You can check with a live-CD after mounting the root filesystem. EDIT: the boot partition of course. You might find it with lsblk -o NAME,FSTYPE,SIZE

What if you press the down arrow over and over while the motherboard logo is shown (should help in case the GRUB menu is just displayed too late).

 

[Der Harry]

Maybe I didnt explain myself correctly, I have booted 8.2 proxmox iso, launched the install and then CTRL + ALT + F3 gets you to the shell, and there i am

chroot, pin the kernel

 

[MrnMrk]

Did you ever modify the timeout/menu settings in /boot/grub/grub.cfg? It should contain an entry

No, I never ever modified anything in the boot section. Infact now that i remember, i never seen the GRUB boot menu in my proxmox server

 

[Der Harry]

Maybe I didnt explain myself correctly, I have booted 8.2 proxmox iso, launched the install and then CTRL + ALT + F3 gets you to the shell, and there i am

Follow the steps here: https://pve.proxmox.com/wiki/Recover_From_Grub_Failure

at the:

Then update grub and install it.

update-grub

Pin the kernel - following my instructions before

(If you had a 6.5 in the past, you simply can chose that at grub and boot - one time - in 6.5)

 

[MrnMrk]

Follow the steps here: https://pve.proxmox.com/wiki/Recover_From_Grub_Failure

at the:

Then update grub and install it.

update-grub

Pin the kernel - following my instructions before

(If you had a 6.5 in the past, you simply can chose that at grub and boot - one time - in 6.5)

Yes, i am following those steps but i am stuck at the: mount /dev/sda1 /media/RESCUE/boot
I dont know what /dev/sda1 is, thats the question.

 

[fiona]

What does lsblk -o NAME,FSTYPE,SIZE show?

 

[Der Harry]

Yes, i am following those steps but i am stuck at the: mount /dev/sda1 /media/RESCUE/boot
I dont know what /dev/sda1 is, thats the question.

cat /proc/partitions

^^^ this shows all paritions

 

[MrnMrk]

cat /proc/partitions

^^^ this shows all paritions

maybe i am not explaining myself. I do not know which partition sda1 the guide is refering to.

 

[MrnMrk]

What does lsblk -o NAME,FSTYPE,SIZE show?

 

[Der Harry]

maybe i am not explaining myself. I do not know which partition sda1 the guide is refering to.

The partition containing the /boot

> sudo mount /dev/sda1 /media/RESCUE/boot <

Basically you can't do anything wrong

You will find some kernels here - if it's not /dev/sda1 maybe /dev/nvmeSOMETHING ...

# ls  /media/RESCUE/boot
System.map-6.5.13-5-pve  config-6.5.13-5-pve  grub                     initrd.img-6.8.4-3-pve  vmlinuz-6.8.4-2-pve
System.map-6.8.4-2-pve   config-6.8.4-2-pve   initrd.img-6.5.13-5-pve  pve                     vmlinuz-6.8.4-3-pve
System.map-6.8.4-3-pve   config-6.8.4-3-pve   initrd.img-6.8.4-2-pve   vmlinuz-6.5.13-5-pve

if you plan to use the pin, you might need to mount also more parts - other then the boot - but then the grub rescue is not the 100% correct approach to follow.

The boot loader grub is working you showed a video. What is not working the kernel.

 

[Der Harry]

Btw. did you try selecting the 6.5.x Kernel?

You can do that at boot time before messing around in the super advanced things....