On a host with ZFS root (rpool on NVMe), UEFI boot via GRUB managed by proxmox-boot-tool, and many NVMe devices (HPE DL360 Gen11, boot device HPE NS204i-u plus 8 data NVMe),
Cause
With ZFS root, grub-mkconfig has no filesystem UUID, so the menu entry IDs contain the device path, e.g.:
The NVMe enumeration order is not stable across boots (the boot disk was nvme0n1 on one boot and nvme4n1 on another). As a result, the GRUB_DEFAULT written by the pin and the menu entry IDs in grub.cfg on the ESP no longer match after a refresh or reboot:
GRUB does not find the ID and silently falls back to entry 0, i.e. the newest kernel. There is no warning, so the user believes the kernel is pinned. In our case this booted the affected kernel again during an incident (bnxt_en firmware crash on 7.0.14-20, see this thread).
Workaround
Set GRUB_DEFAULT by menu title, which does not depend on device names:
The
Suggestion
For GRUB, generate the pin by menu title, or use a stable identifier (e.g. pool GUID or a by-id path) instead of /dev/nvmeXnY.
Versions: pve-manager 9.2.21, proxmox-kernel-helper 9.2.0, GRUB 2.12-9+pmx2
proxmox-boot-tool kernel pin has no effect.Cause
With ZFS root, grub-mkconfig has no filesystem UUID, so the menu entry IDs contain the device path, e.g.:
Code:
gnulinux-advanced-/dev/nvme0n1p3>gnulinux-7.0.14-17-pve-advanced-/dev/nvme0n1p3The NVMe enumeration order is not stable across boots (the boot disk was nvme0n1 on one boot and nvme4n1 on another). As a result, the GRUB_DEFAULT written by the pin and the menu entry IDs in grub.cfg on the ESP no longer match after a refresh or reboot:
Code:
/etc/default/grub.d/proxmox-kernel-pin.cfg:
GRUB_DEFAULT="gnulinux-advanced-/dev/nvme0n1p3>gnulinux-7.0.14-17-pve-advanced-/dev/nvme0n1p3"
ESP grub/grub.cfg:
set default="gnulinux-advanced-/dev/nvme0n1p3>gnulinux-7.0.14-17-pve-advanced-/dev/nvme0n1p3"
menuentry_id_option 'gnulinux-advanced-/dev/nvme4n1p3'
menuentry_id_option 'gnulinux-7.0.14-17-pve-advanced-/dev/nvme4n1p3'GRUB does not find the ID and silently falls back to entry 0, i.e. the newest kernel. There is no warning, so the user believes the kernel is pinned. In our case this booted the affected kernel again during an incident (bnxt_en firmware crash on 7.0.14-20, see this thread).
Workaround
Set GRUB_DEFAULT by menu title, which does not depend on device names:
Code:
cat > /etc/default/grub.d/zz-kernel-pin-title.cfg <<'EOF'
GRUB_DEFAULT="Advanced options for Proxmox VE GNU/Linux>Proxmox VE GNU/Linux, with Linux 7.0.14-17-pve"
EOF
proxmox-boot-tool refreshzz- prefix makes it override proxmox-kernel-pin.cfg. Remember to remove it when unpinning.Suggestion
For GRUB, generate the pin by menu title, or use a stable identifier (e.g. pool GUID or a by-id path) instead of /dev/nvmeXnY.
Versions: pve-manager 9.2.21, proxmox-kernel-helper 9.2.0, GRUB 2.12-9+pmx2