System details:
Timeline:
Packages upgraded in the failing transaction (from /var/log/apt/history.log):
Evidence collected post-recovery:
Recovery action: Powered off, disconnected SATA power and data cables from the OS SSD, system booted to BIOS, powered back off, plugged OS SSD back in, flipped PSU on, system booted and the GUI was reachable. . System came up cleanly on the first attempt and has been stable since.
Hypothesis: The GRUB EFI package upgrade on a Secure Boot-enabled Proxmox 9 system left the UEFI firmware in an inconsistent cached state related to the Secure Boot chain or NVRAM device binding. The ESP files themselves were installed correctly (md5sums match the signed binary on disk, timestamps consistent with the apt transaction). The failure was pre-kernel — zero journal entries for 15+ hours of attempted boots. Forcing SATA re-enumeration by disconnecting and reconnecting the OS SSD cleared whatever firmware state was stuck and allowed the Secure Boot chain to execute cleanly on the next attempt.
I can’t definitively say whether the stuck state was a cached validation hash, a shim-to-GRUB handoff memory artifact, or stale NVRAM device tracking — only that a SATA-level device re-enumeration fixed it, which implicates firmware state rather than file content.
I must also state that I had no monitor hooked up during the failing boot attempts — this is a headless Proxmox node and my first instinct was to troubleshoot by LED pattern and timing rather than to attach a display. In hindsight I should have, but by the time I had reason to believe the issue was pre-kernel rather than hardware-side, I had already recovered via SATA re-enumeration and could no longer reproduce the failure to capture on-screen output. So I cannot report whether a Security Violation 0x1A, a shim MOK dialog, a GRUB rescue prompt, or something else was being displayed at the 16-second mark.
The first time I did have a monitor hooked up was after the OS SSD reseat. At that point the system POSTed and reached the Proxmox login screen normally. No errors on screen, no delays.
I had AI assistance from Claude in walking through the diagnostic steps and collecting the evidence above. Posting here to see if the Proxmox community has seen this pattern and to contribute the recovery procedure (SATA re-enumeration) as a datapoint for anyone else who hits it.
I tried to find similar occurrences on here but have not found anything. A few posts came up on grub related and or kernel topics but they were old. And since I have been stable across three unclustered nodes for over two years, I want to find the issue.
Edit: added my system stability context at the end.
Edit2: proxmox-secure-boot-support has never been installed on this system (install dates to Feb 2025, apt/dpkg history shows no install/remove record for the package). This hasn’t caused problems through many previous dist-upgrades, so I’m not claiming it’s causally related to this failure, just flagging it in case it’s relevant. Would the Proxmox team recommend installing it on an existing Secure-Boot-enabled PVE 9 host? I want to say that I made the PVE 8 > 9 updated on a previous motherboard and CPU (went from i9-9900k > i9-12900k). HAve had many kernel updates and reboots since with 0 stability issues up until this one.
- Proxmox VE 9.x, Boot Mode: EFI (Secure Boot)
- Kernel at time of failure: 7.0.14-28-pve (installed in failing transaction)
- Kernel prior: 7.0.14-26-pve
- Hardware: MSI Pro Z790-P WiFi, Intel i9-12900K, 128GB RAM, RTX 5060 Ti (passthrough), MSI Pro Z790-P WiFi motherboard
- Root: LVM on single SATA SSD (not ZFS; proxmox-boot-tool not managing ESP — single ESP via standard grub-install)
- No manual intervention between update and reboot
Timeline:
- 2026-11-04 18:52:37 — Upgrade from the Web UI completed cleanly at 18:53:34, kernel upgrade so a reboot was needed. See package list below.
- 2026-11-04 23:43:45 — journald recorded the clean shutdown End-Date (last clean state).
- Post-reboot: system failed to boot. Power on, LEDs on MSI EZ Debug cycled Red → Amber → White → Red, hard power-off at ~16 seconds. No display output at any time. Repeatable across many attempts.
- 6+ hours of troubleshooting on misleading hardware-side evidence (aging AIO H100i from 2020, since confirmed independent).
- Disconnected a few HDDs in the system and one SATA SSD (not the OS SSD) to see if system would boot.
- 2026-11-05 ~15:20 — disconnected OS SSD SATA power and data cables, then reconnected. Next boot succeeded. Current uptime stable since.
Packages upgraded in the failing transaction (from /var/log/apt/history.log):
Code:
proxmox-kernel-7.0.14-28-pve-signed (new install, auto)
proxmox-kernel-7.0 7.0.14-26 → 7.0.14-28
grub-efi-amd64-signed 1+2.12+9+pmx2 → 1+2.12+9+pmx4
grub-efi-amd64 2.12-9+pmx2 → 2.12-9+pmx4
grub-efi-amd64-unsigned 2.12-9+pmx2 → 2.12-9+pmx4
grub-efi-amd64-bin 2.12-9+pmx2 → 2.12-9+pmx4
grub-pc-bin 2.12-9+pmx2 → 2.12-9+pmx4
grub-common 2.12-9+pmx2 → 2.12-9+pmx4
grub2-common 2.12-9+pmx2 → 2.12-9+pmx4
proxmox-grub 2.12-9+pmx2 → 2.12-9+pmx4
pve-firmware 3.18-8 → 3.18-9
proxmox-backup-restore-image 1.0.0 → 1.1.0
proxmox-widget-toolkit 5.2.10 → 5.2.11
libpve-rs-perl 0.15.3 → 0.15.4
apt 3.0.3 → 3.0.4
libapt-pkg6.0 2.6.1 → 2.6.2
libapt-pkg7.0 3.0.3 → 3.0.4
apt-utils 3.0.3 → 3.0.4
libpve-guest-common-perl 6.0.5 → 6.0.6
libpve-access-control 9.1.2 → 9.2.0
pve-container 6.1.16 → 6.1.18
libpve-network-api-perl 1.6.7 → 1.6.8
libpve-network-perl 1.6.7 → 1.6.8
libpve-http-server-perl 6.0.5 → 6.0.6
libpve-common-perl 9.2.6 → 9.2.7
proxmox-firewall 1.2.3 → 1.2.4
pve-firewall 6.0.6 → 6.0.7
ifupdown2 3.3.0-1+pmx12 → 3.3.0-1+pmx13Evidence collected post-recovery:
- journalctl --list-boots — zero boot entries exist between the clean shutdown at 2026-11-04 23:43:45 and the successful boot at 2026-11-05 15:27:52. 15+ hours of failed boot attempts produced no journal entries at all, indicating the kernel never reached journald. Failure was pre-journald, meaning early-init or pre-GRUB.
- efibootmgr -v — single real boot entry (Boot0001 debian → \EFI\debian\shimx64.efi), Timeout: 0, no fallback entries configured. Secure Boot chain: shim → GRUB → kernel.
- proxmox-boot-tool status — “/etc/kernel/proxmox-boot-uuids does not exist.” ESP is managed by stock grub-install, not proxmox-boot-tool.
- md5sum on ESP grubx64.efi vs /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed:
-
Code:
[root@i9 ~]# ls -la /boot/efi/EFI/debian/ && md5sum /boot/efi/EFI/debian/grubx64.efi /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed total 4624 drwxr-xr-x 2 root root 4096 Nov 4 18:53 . drwxr-xr-x 5 root root 4096 Nov 4 18:53 .. -rwxr-xr-x 1 root root 112 Nov 4 18:53 BOOTX64.CSV -rwxr-xr-x 1 root root 120024 Nov 4 18:53 fbx64.efi -rwxr-xr-x 1 root root 201 Nov 4 18:53 grub.cfg -rwxr-xr-x 1 root root 2685544 Nov 4 18:53 grubx64.efi -rwxr-xr-x 1 root root 875096 Nov 4 18:53 mmx64.efi -rwxr-xr-x 1 root root 1032032 Nov 4 18:53 shimx64.efi c9355d4fb25202f74af048c6956fe620 /boot/efi/EFI/debian/grubx64.efi c9355d4fb25202f74af048c6956fe620 /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed [root@i9 ~]# file /boot/efi/EFI/debian/shimx64.efi /boot/efi/EFI/debian/grubx64.efi
Recovery action: Powered off, disconnected SATA power and data cables from the OS SSD, system booted to BIOS, powered back off, plugged OS SSD back in, flipped PSU on, system booted and the GUI was reachable. . System came up cleanly on the first attempt and has been stable since.
Hypothesis: The GRUB EFI package upgrade on a Secure Boot-enabled Proxmox 9 system left the UEFI firmware in an inconsistent cached state related to the Secure Boot chain or NVRAM device binding. The ESP files themselves were installed correctly (md5sums match the signed binary on disk, timestamps consistent with the apt transaction). The failure was pre-kernel — zero journal entries for 15+ hours of attempted boots. Forcing SATA re-enumeration by disconnecting and reconnecting the OS SSD cleared whatever firmware state was stuck and allowed the Secure Boot chain to execute cleanly on the next attempt.
I can’t definitively say whether the stuck state was a cached validation hash, a shim-to-GRUB handoff memory artifact, or stale NVRAM device tracking — only that a SATA-level device re-enumeration fixed it, which implicates firmware state rather than file content.
I must also state that I had no monitor hooked up during the failing boot attempts — this is a headless Proxmox node and my first instinct was to troubleshoot by LED pattern and timing rather than to attach a display. In hindsight I should have, but by the time I had reason to believe the issue was pre-kernel rather than hardware-side, I had already recovered via SATA re-enumeration and could no longer reproduce the failure to capture on-screen output. So I cannot report whether a Security Violation 0x1A, a shim MOK dialog, a GRUB rescue prompt, or something else was being displayed at the 16-second mark.
The first time I did have a monitor hooked up was after the OS SSD reseat. At that point the system POSTed and reached the Proxmox login screen normally. No errors on screen, no delays.
I had AI assistance from Claude in walking through the diagnostic steps and collecting the evidence above. Posting here to see if the Proxmox community has seen this pattern and to contribute the recovery procedure (SATA re-enumeration) as a datapoint for anyone else who hits it.
I tried to find similar occurrences on here but have not found anything. A few posts came up on grub related and or kernel topics but they were old. And since I have been stable across three unclustered nodes for over two years, I want to find the issue.
Edit: added my system stability context at the end.
Edit2: proxmox-secure-boot-support has never been installed on this system (install dates to Feb 2025, apt/dpkg history shows no install/remove record for the package). This hasn’t caused problems through many previous dist-upgrades, so I’m not claiming it’s causally related to this failure, just flagging it in case it’s relevant. Would the Proxmox team recommend installing it on an existing Secure-Boot-enabled PVE 9 host? I want to say that I made the PVE 8 > 9 updated on a previous motherboard and CPU (went from i9-9900k > i9-12900k). HAve had many kernel updates and reboots since with 0 stability issues up until this one.
Last edited:
