[solved] pve 8.0.4 no boot from system disk after kernel update

P

Peter Altherr

Renowned Member
Apr 28, 2016
28
5
68
56
dear proxmox experts, today i did the proxmox kernel update via webui from 6.2.16-10 to 6.2.16-14. after reboot my hp microserver does no longer found its boot media. i downloaded the most current pve install iso 8.0-2 and did a rescue boot. now i have my pve installation back to life again :) unfortunately i have no idea how to proceed in order to fix the boot problem on the system disk. what would be the next steps in order to get it boot? i have found some threads in the forum about how to fix boot problems where at least grub boot loader is getting loaded. but in my case the problem is the server does not even recognize the boot media as bootable. any clue is highly appreciated.
 
Last edited:
Hi, if you are running PVE 8, is it a fresh installation or did you upgrade from PVE 7 at some point? To find out whether you are affected by the grub bug that you are probably referring to [1], could you run the following commands and post their output?

Code: 
pveversion -v
efibootmgr -v
vgscan -vvv 2>&1 | grep metadata

[1] https://forum.proxmox.com/threads/error-disk-lvmid-not-found-grub-rescue.123512/
 
hey friedrich, thanks for guiding me. the original installation is indeed a pve 7. right after the release of pve 8 i did an upgrade according the proxmox wiki. everything went well without any error messages. since then i had several kernel updates but this is the first resulting in an unbootable boot device :)

heres the output of the given 3 commands:

Bash: 
root@pve:~# pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-14-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
pve-kernel-5.15: 7.4-4
proxmox-kernel-6.2.16-14-pve: 6.2.16-14
proxmox-kernel-6.2: 6.2.16-14
proxmox-kernel-6.2.16-12-pve: 6.2.16-12
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.8
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

Bash: 
root@pve:~# efibootmgr -v
EFI variables are not supported on this system.

Bash: 
root@pve:~# vgscan -vvv 2>&1 | grep metadata
  metadata/record_lvs_history not found in config: defaulting to 0
  File locking settings: readonly:0 sysinit:0 ignorelockingfailure:0 global/metadata_read_only:0 global/wait_for_locks:1.
  Reading metadata summary from /dev/sde3 at 28160 size 2560 (+0)
  Found metadata summary on /dev/sde3 at 28160 size 2560 for VG pve
  Found metadata seqno 11 in mda1 on /dev/sde3
  Reading VG pve metadata from /dev/sde3 4096
  Reading metadata from /dev/sde3 at 28160 size 2560 (+0)
  Logical volume pve/lvol0_pmspare is pool metadata spare.
  Found metadata text at 28160 off 24064 size 2560 VG pve on /dev/sde3
  metadata/lvs_history_retention_time not found in config: defaulting to 0
  Found volume group "pve" using metadata type lvm2
 
Hi, thanks for the information. The grub bug is fixed in the version shipped with PVE 8, which you are running, but currently it can still trigger on systems booting in UEFI mode that were upgraded from PVE 7, see [1]. According to the efibootmgr output however, the machine is not booted in UEFI mode, so it should not be affected. So the issue seems to be something different.

First, to rule out the easy cases, could you double-check that the disk is included in the boot order configured in the BIOS?

If this looks good, maybe something went wrong after the last kernel upgrade. Could you open /var/log/apt/term.log, find the entries (starting with Log started: and ending with Log ended:) of the last few upgrades and post them here?

[1] https://forum.proxmox.com/threads/error-disk-lvmid-not-found-grub-rescue.123512/post-591780
 
in fact i wonder a little why it reports currently not to be booted by uefi, as the partition structure indicates an uefi installation:
1695914738444.png

but the bios of this generation of hp microservers (ML10v2) is more than a little bitchy. i spent hours in order to find out it will only boot from a bootable device connected to the first sata port of the onboard controller no matter what is configured in bios. i did not change bios configuration nor connecting the ssd to a different port after the kernel update. anyway i am going to look into bios later this evening as the machine is in use right now :)

please find here the content of /var/log/apt/term.log:

Bash: 
Log started: 2023-09-07  22:27:30
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 70082 files and directories currently installed.)
Preparing to unpack .../pve-firmware_3.8-2_all.deb ...
Unpacking pve-firmware (3.8-2) over (3.7-1) ...
Selecting previously unselected package proxmox-kernel-6.2.16-12-pve.
Preparing to unpack .../proxmox-kernel-6.2.16-12-pve_6.2.16-12_amd64.deb ...
Unpacking proxmox-kernel-6.2.16-12-pve (6.2.16-12) ...
Preparing to unpack .../proxmox-kernel-6.2_6.2.16-12_all.deb ...
Unpacking proxmox-kernel-6.2 (6.2.16-12) over (6.2.16-10) ...
Preparing to unpack .../pve-qemu-kvm_8.0.2-5_amd64.deb ...
Unpacking pve-qemu-kvm (8.0.2-5) over (8.0.2-4) ...
Preparing to unpack .../qemu-server_8.0.7_amd64.deb ...
Unpacking qemu-server (8.0.7) over (8.0.6) ...
Setting up proxmox-kernel-6.2.16-12-pve (6.2.16-12) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.2.16-12-pve /boot/vmlinuz-6.2.16-12-pve
update-initramfs: Generating /boot/initrd.img-6.2.16-12-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 6.2.16-12-pve /boot/vmlinuz-6.2.16-12-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 6.2.16-12-pve /boot/vmlinuz-6.2.16-12-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.2.16-12-pve /boot/vmlinuz-6.2.16-12-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.2.16-12-pve
Found initrd image: /boot/initrd.img-6.2.16-12-pve
Found linux image: /boot/vmlinuz-6.2.16-10-pve
Found initrd image: /boot/initrd.img-6.2.16-10-pve
Found linux image: /boot/vmlinuz-6.2.16-8-pve
Found initrd image: /boot/initrd.img-6.2.16-8-pve
Found linux image: /boot/vmlinuz-5.15.108-1-pve
Found initrd image: /boot/initrd.img-5.15.108-1-pve
Found linux image: /boot/vmlinuz-5.15.102-1-pve
Found initrd image: /boot/initrd.img-5.15.102-1-pve
Found memtest86+x64 image: /boot/memtest86+x64.bin
done
Setting up pve-qemu-kvm (8.0.2-5) ...
Setting up pve-firmware (3.8-2) ...
Setting up qemu-server (8.0.7) ...
Setting up proxmox-kernel-6.2 (6.2.16-12) ...
Processing triggers for pve-ha-manager (4.0.2) ...
Processing triggers for pve-manager (8.0.4) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for mailcap (3.70+nmu1) ...
Log ended: 2023-09-07  22:28:12

Log started: 2023-09-27  11:41:54
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 77599 files and directories currently installed.)
Preparing to unpack .../0-bind9-host_1%3a9.18.19-1~deb12u1_amd64.deb ...
Unpacking bind9-host (1:9.18.19-1~deb12u1) over (1:9.18.16-1~deb12u1) ...
Preparing to unpack .../1-bind9-dnsutils_1%3a9.18.19-1~deb12u1_amd64.deb ...
Unpacking bind9-dnsutils (1:9.18.19-1~deb12u1) over (1:9.18.16-1~deb12u1) ...
Preparing to unpack .../2-bind9-libs_1%3a9.18.19-1~deb12u1_amd64.deb ...
Unpacking bind9-libs:amd64 (1:9.18.19-1~deb12u1) over (1:9.18.16-1~deb12u1) ...
Selecting previously unselected package proxmox-kernel-6.2.16-14-pve.
Preparing to unpack .../3-proxmox-kernel-6.2.16-14-pve_6.2.16-14_amd64.deb ...
Unpacking proxmox-kernel-6.2.16-14-pve (6.2.16-14) ...
Preparing to unpack .../4-proxmox-kernel-6.2_6.2.16-14_all.deb ...
Unpacking proxmox-kernel-6.2 (6.2.16-14) over (6.2.16-12) ...
Preparing to unpack .../5-pve-i18n_3.0.7_all.deb ...
Unpacking pve-i18n (3.0.7) over (3.0.5) ...
Preparing to unpack .../6-pve-qemu-kvm_8.0.2-6_amd64.deb ...
Unpacking pve-qemu-kvm (8.0.2-6) over (8.0.2-5) ...
Setting up bind9-libs:amd64 (1:9.18.19-1~deb12u1) ...
Setting up pve-qemu-kvm (8.0.2-6) ...
Setting up pve-i18n (3.0.7) ...
Setting up proxmox-kernel-6.2.16-14-pve (6.2.16-14) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.2.16-14-pve /boot/vmlinuz-6.2.16-14-pve
update-initramfs: Generating /boot/initrd.img-6.2.16-14-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 6.2.16-14-pve /boot/vmlinuz-6.2.16-14-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 6.2.16-14-pve /boot/vmlinuz-6.2.16-14-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.2.16-14-pve /boot/vmlinuz-6.2.16-14-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.2.16-14-pve
Found initrd image: /boot/initrd.img-6.2.16-14-pve
Found linux image: /boot/vmlinuz-6.2.16-12-pve
Found initrd image: /boot/initrd.img-6.2.16-12-pve
Found linux image: /boot/vmlinuz-6.2.16-10-pve
Found initrd image: /boot/initrd.img-6.2.16-10-pve
Found linux image: /boot/vmlinuz-6.2.16-8-pve
Found initrd image: /boot/initrd.img-6.2.16-8-pve
Found linux image: /boot/vmlinuz-5.15.108-1-pve
Found initrd image: /boot/initrd.img-5.15.108-1-pve
Found linux image: /boot/vmlinuz-5.15.102-1-pve
Found initrd image: /boot/initrd.img-5.15.102-1-pve
Found memtest86+x64 image: /boot/memtest86+x64.bin
done
Setting up bind9-host (1:9.18.19-1~deb12u1) ...
Setting up proxmox-kernel-6.2 (6.2.16-14) ...
Setting up bind9-dnsutils (1:9.18.19-1~deb12u1) ...
Processing triggers for libc-bin (2.36-9+deb12u1) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for mailcap (3.70+nmu1) ...
Log ended: 2023-09-27  11:42:23

Log started: 2023-09-27  20:43:41
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 85097 files and directories currently installed.)
Removing proxmox-kernel-6.2.16-10-pve (6.2.16-10) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 6.2.16-10-pve /boot/vmlinuz-6.2.16-10-pve
update-initramfs: Deleting /boot/initrd.img-6.2.16-10-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 6.2.16-10-pve /boot/vmlinuz-6.2.16-10-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 6.2.16-10-pve /boot/vmlinuz-6.2.16-10-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 6.2.16-10-pve /boot/vmlinuz-6.2.16-10-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.2.16-14-pve
Found initrd image: /boot/initrd.img-6.2.16-14-pve
Found linux image: /boot/vmlinuz-6.2.16-12-pve
Found initrd image: /boot/initrd.img-6.2.16-12-pve
Found linux image: /boot/vmlinuz-6.2.16-8-pve
Found initrd image: /boot/initrd.img-6.2.16-8-pve
Found linux image: /boot/vmlinuz-5.15.108-1-pve
Found initrd image: /boot/initrd.img-5.15.108-1-pve
Found linux image: /boot/vmlinuz-5.15.102-1-pve
Found initrd image: /boot/initrd.img-5.15.102-1-pve
Found memtest86+x64 image: /boot/memtest86+x64.bin
done
Removing proxmox-kernel-6.2.16-8-pve (6.2.16-8) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 6.2.16-8-pve /boot/vmlinuz-6.2.16-8-pve
update-initramfs: Deleting /boot/initrd.img-6.2.16-8-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 6.2.16-8-pve /boot/vmlinuz-6.2.16-8-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 6.2.16-8-pve /boot/vmlinuz-6.2.16-8-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 6.2.16-8-pve /boot/vmlinuz-6.2.16-8-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.2.16-14-pve
Found initrd image: /boot/initrd.img-6.2.16-14-pve
Found linux image: /boot/vmlinuz-6.2.16-12-pve
Found initrd image: /boot/initrd.img-6.2.16-12-pve
Found linux image: /boot/vmlinuz-5.15.108-1-pve
Found initrd image: /boot/initrd.img-5.15.108-1-pve
Found linux image: /boot/vmlinuz-5.15.102-1-pve
Found initrd image: /boot/initrd.img-5.15.102-1-pve
Found memtest86+x64 image: /boot/memtest86+x64.bin
done
Log ended: 2023-09-27  20:43:53
 
after digging into my bios settings i guess i found the root cause for not booting after kernel upgrade however i have no idea how it occured. hp microserver bios requires to define a boot controller where (as i wrote earlier) a bootable drive is supposed to be connected on its first port. the bios here reports 3 mass storage controllers where the onboard controller slipped from first position (boot controller) down to second. i put it back to first position, reboot and pve is up and running again :)

the only thing i have to investigate: while watching the boot process i got noticed of two lines regarding my zfs pools where i am going to open a new thread.

thanks a lot for supporting me.
 
  • Like
Reactions: fweber
Same here. What a nightmare. I updated my proxmox server from almost the latest version to the version from today and after a reboot the server was not able to boot anymore.

Here is my logfile:

I was able to solve the problem by connection my headless server with a monitor and keyboard. I logged in to the terminal and installed the grub package via apt install grub-efi-amd64

My error was:
"System booted in EFI-mode but 'grub-efi-amd64' meta-package not installed!"

https://forum.proxmox.com/threads/u...-efi-amd64-meta-package-not-installed.137324/

Now the server is back again and working. But I had some hours to investigate into this error!.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom