vmlinuz not found, you need to run kernel first

[IsThisThingOn]

Seems like I somehow destroyed my Proxmox installation, hopefully someone can help me restore it
Today I ran all the latest updates, shutdown the system and did a BIOS update.

When it loads up GRUB, I see 3 different version to boot
Proxmox Virtual Environment GNU/Linux, with Linux 5.4.128-1-pve
Proxmox Virtual Environment GNU/Linux, with Linux 5.4.124-1-pve
Proxmox Virtual Environment GNU/Linux, with Linux 5.3.18-3-pve

Every one of them gives me the same error.

Loading Linux 5.4.128-1-pve...
error: file (vmlinuz-5.4.128-1-pve' not found.
Loading inital ramdisk ...
error: you need to load the kernel first.

Any idea how I can safe this installation?

The installation is on a nvme ZFS mirror.

 

[IsThisThingOn]

Update: Well, that was a stupid mistake. The BIOS update changed the boot order back, so it used legacy instead of UEFI.

 

[MasterCATZ]

in my case I only have legacy mode assuming when I upgraded from 8 to 8.2 it installed UEFI and now I am unable to boot ?

Loading Linux 6.8.8-2-pve...
error: invalid cluster 0
Loading inital ramdisk ...
error: you need to load the kernel first.



now trying to repair from live boot with chroot now having to work out why it's saying ReadOnly ,
I can create and save files in /var/tmp/espmounts/43E5-6E0F

# mount -t proc proc /mnt/rpool/pve-1/proc
root@debian:/mnt/rpool/pve-1# mount -t sysfs sys /mnt/rpool/pve-1/sys
root@debian:/mnt/rpool/pve-1# mount -o bind /dev /mnt/rpool/pve-1/dev
mount none -t devpts /dev/pts



oot@debian:/# apt full-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be REMOVED:
pve-kernel-5.15.64-1-pve
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 400 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 161496 files and directories currently installed.)
Removing pve-kernel-5.15.64-1-pve (5.15.64-1) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.15.64-1-pve /boot/vmlinuz-5.15.64-1-pve
update-initramfs: Deleting /boot/initrd.img-5.15.64-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.15.64-1-pve /boot/vmlinuz-5.15.64-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.15.64-1-pve /boot/vmlinuz-5.15.64-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/43E5-6E0F
Copying kernel 6.2.16-20-pve
cp: preserving times for '/var/tmp/espmounts/43E5-6E0F/initrd.img-6.2.16-20-pve': Read-only file system
run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.15.64-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.15.64-1-pve (--remove):
installed pve-kernel-5.15.64-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
pve-kernel-5.15.64-1-pve
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)


apt install proxmox-default-kernel -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
proxmox-default-kernel is already the newest version (1.1.0).
proxmox-default-kernel set to manually installed.
The following package was automatically installed and is no longer required:
proxmox-kernel-6.2.16-18-pve
Use 'apt autoremove' to remove it.
The following packages will be REMOVED:
pve-kernel-5.15.64-1-pve
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 400 MB disk space will be freed.
(Reading database ... 161496 files and directories currently installed.)
Removing pve-kernel-5.15.64-1-pve (5.15.64-1) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.15.64-1-pve /boot/vmlinuz-5.15.64-1-pve
update-initramfs: Deleting /boot/initrd.img-5.15.64-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.15.64-1-pve /boot/vmlinuz-5.15.64-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.15.64-1-pve /boot/vmlinuz-5.15.64-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/43E5-6E0F
Copying kernel 6.2.16-20-pve
cp: preserving times for '/var/tmp/espmounts/43E5-6E0F/vmlinuz-6.2.16-20-pve': Read-only file system
run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.15.64-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.15.64-1-pve (--remove):
installed pve-kernel-5.15.64-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
pve-kernel-5.15.64-1-pve
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)


debian live 12.5 bookworm user/pass = user/live
apt update
apt install openssh-server

sed -r -i'.BAK' 's/^deb(.*)$/deb\1 contrib/g' /etc/apt/sources.list
apt update
apt install linux-headers-amd64 zfsutils-linux zfs-dkms zfs-zed
modprobe zfs

zpool import -f -R /mnt rpool
mount -o rbind /proc /mnt/proc
mount -o rbind /sys /mnt/sys
mount -o rbind /dev /mnt/dev
mount -o rbind /run /mnt/run
chroot /mnt /bin/bash

got me no where using debian live


*EDIT*
Ended up using proxmox ISO in debug mode and pressing Ctrl + D

lsblk
to find the 512m partition /dev/sda2

zpool import -f -R /mnt rpool
mount -o rbind /proc /mnt/proc
mount -o rbind /sys /mnt/sys
mount -o rbind /dev /mnt/dev
mount -o rbind /run /mnt/run
chroot /mnt /bin/bash

proxmox-boot-tool format /dev/sda2 --force
proxmox-boot-tool init /dev/sda2
proxmox-boot-tool clean
proxmox-boot-tool status


initramfs dropped to its shell
zpool import -f rpool
rebooted
finally booting