Proxmox 7.2 EFI/ZFS Default Kernel Boot

Cataract2

New Member
Jan 16, 2021
12
0
1
41
I currently have Proxmox 7.2 installed with EFI boot and ZFS. So, my system utilizes systemd for boot which means changes in grub do nothing. I'm stuck on how I change the default kernel, which right now is 5.15 for Proxmox 7.2, back to 5.13 as I need that kernel for now so I can utilize Nvidia vGPU drivers to split the gpu among my VMs. I know I can manually select 5.13 during boot, but as this is a headless system that is not ideal.
 
Yeah, as the docs state you need to make the changes live by executing proxmox-boot-tool refresh
So I followed the doc and tried the proxmox-boot-tool kernel pin option to select the 5.13 kernel. I ran the refresh command with it and yet upon reboot I am seeing it select the 5.15 kernel still.
1653602058219.png
 
So, it went something like this?

Code:
root@pve:~# proxmox-boot-tool kernel pin 5.13.19-6-pve
Set kernel '5.13.19-6-pve' in /etc/kernel/proxmox-boot-pin.
Refresh the actual boot ESPs now? [yN] y
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/9846-C1B1
        Copying kernel and creating boot-entry for 5.13.19-6-pve
        Copying kernel and creating boot-entry for 5.15.30-2-pve
        Copying kernel and creating boot-entry for 5.15.35-1-pve
root@pve:~#
 
So, it went something like this?

Code:
root@pve:~# proxmox-boot-tool kernel pin 5.13.19-6-pve
Set kernel '5.13.19-6-pve' in /etc/kernel/proxmox-boot-pin.
Refresh the actual boot ESPs now? [yN] y
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/9846-C1B1
        Copying kernel and creating boot-entry for 5.13.19-6-pve
        Copying kernel and creating boot-entry for 5.15.30-2-pve
        Copying kernel and creating boot-entry for 5.15.35-1-pve
root@pve:~#
Yes. It seems strange that it's not taking affect. Like it's not actually changing anything.
root@pve:~# proxmox-boot-tool kernel pin 5.13.19-6-pve
Set kernel '5.13.19-6-pve' in /etc/kernel/proxmox-boot-pin.
Refresh the actual boot ESPs now? [yN] y
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/E89E-57D3
Copying kernel and creating boot-entry for 5.13.19-6-pve
Copying kernel and creating boot-entry for 5.15.35-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/E89F-21C3
Copying kernel and creating boot-entry for 5.13.19-6-pve
Copying kernel and creating boot-entry for 5.15.35-1-pve
root@pve:~# proxmox-boot-tool refresh
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/E89E-57D3
Copying kernel and creating boot-entry for 5.13.19-6-pve
Copying kernel and creating boot-entry for 5.15.35-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/E89F-21C3
Copying kernel and creating boot-entry for 5.13.19-6-pve
Copying kernel and creating boot-entry for 5.15.35-1-pve
 
Yes. It seems strange that it's not taking affect. Like it's not actually changing anything.
the change is in the loader.conf - mount the ESP (e.g. /dev/disk/by-uuid/E89F-21C3) and check $ESPMOUNTPOINT/loader/loader.conf
it should be the same on all your ESPs.

also make sure you are actually booting from one of the listed ESPs (and don't have some other ESP in the system)

I hope this helps!
 
the change is in the loader.conf - mount the ESP (e.g. /dev/disk/by-uuid/E89F-21C3) and check $ESPMOUNTPOINT/loader/loader.conf
it should be the same on all your ESPs.

also make sure you are actually booting from one of the listed ESPs (and don't have some other ESP in the system)

I hope this helps!
So, I mounted the ESP and checked the loader.conf. I show the kernel I pined in there.
1653697407679.png

So it is putting it in there. So I moved to the next part about which ESP might be in there. I do show both drives and I checks the ESPs on both in the loader.conf and both show the same as above.
1653697480017.png
Yet the boot still chooses the 5.15.
 
Last edited:
Yet the boot still chooses the 5.15.
any chance you've set a version to be booted in an efi-var ? (usually with `bootctl` or inside the BIOS)?

else could you please post the complete output of `lsblk` (as text inside code tags)
 
any chance you've set a version to be booted in an efi-var ? (usually with `bootctl` or inside the BIOS)?

else could you please post the complete output of `lsblk` (as text inside code tags)
I don't believe I did. But here are the outputs of both bootctl and lsblk. Bios is set to just point to one of the two boot drives on booting.

Code:
root@pve:~# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0    50G  0 loop
loop1    7:1    0    60G  0 loop
sda      8:0    0   5.5T  0 disk
├─sda1   8:1    0   5.5T  0 part
└─sda9   8:9    0     8M  0 part
sdb      8:16   0   5.5T  0 disk
├─sdb1   8:17   0   5.5T  0 part
└─sdb9   8:25   0     8M  0 part
sdc      8:32   0   5.5T  0 disk
├─sdc1   8:33   0   5.5T  0 part
└─sdc9   8:41   0     8M  0 part
sdd      8:48   0   5.5T  0 disk
├─sdd1   8:49   0   5.5T  0 part
└─sdd9   8:57   0     8M  0 part
sde      8:64   0   5.5T  0 disk
├─sde1   8:65   0   5.5T  0 part
└─sde9   8:73   0     8M  0 part
sdf      8:80   0   1.8T  0 disk
├─sdf1   8:81   0   1.8T  0 part
└─sdf9   8:89   0     8M  0 part
sdg      8:96   0   1.8T  0 disk
├─sdg1   8:97   0   1.8T  0 part
└─sdg9   8:105  0     8M  0 part
sdh      8:112  0   1.8T  0 disk
├─sdh1   8:113  0   1.8T  0 part
└─sdh9   8:121  0     8M  0 part
sdi      8:128  0   1.8T  0 disk
├─sdi1   8:129  0   1.8T  0 part
└─sdi9   8:137  0     8M  0 part
sdj      8:144  0 465.8G  0 disk
├─sdj1   8:145  0  1007K  0 part
├─sdj2   8:146  0   512M  0 part
└─sdj3   8:147  0 465.3G  0 part
sdk      8:160  0 465.8G  0 disk
├─sdk1   8:161  0  1007K  0 part
├─sdk2   8:162  0   512M  0 part
└─sdk3   8:163  0 465.3G  0 part
sr0     11:0    1  1024M  0 rom

Code:
root@pve:~# bootctl
Couldn't find EFI system partition. It is recommended to mount it to /boot or /efi.
Alternatively, use --esp-path= to specify path to mount point.
System:
     Firmware: UEFI 2.70 (American Megatrends 5.17)
  Secure Boot: disabled
   Setup Mode: setup
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 241
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✗ Support for XBOOTLDR partition
               ✗ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
WARNING: The boot loader reports different ESP UUID then detected!
          ESP: /dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3
         File: └─/EFI/BOOT/BOOTX64.EFI

Random Seed:
 Passed to OS: no
 System Token: not set

Boot Loaders Listed in EFI Variables:
        Title: UEFI OS
           ID: 0x0007
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3
         File: └─/EFI/BOOT/BOOTX64.EFI

        Title: UEFI OS
           ID: 0x0008
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/2c464620-9669-44c3-985f-f9a1505fab4d
         File: └─/EFI/BOOT/BOOTX64.EFI
 
could you try mounting and checking those 2 partitions?
Certainly, first one is E89F-21C3.
Code:
root@pve:/# mount /dev/disk/by-uuid/E89F-21C3 /boot/efi/
root@pve:/# bootctl
System:
     Firmware: UEFI 2.70 (American Megatrends 5.17)
  Secure Boot: disabled
   Setup Mode: setup
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 241
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✗ Support for XBOOTLDR partition
               ✗ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
          ESP: /dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3
         File: └─/EFI/BOOT/BOOTX64.EFI

Random Seed:
 Passed to OS: no
 System Token: not set
       Exists: no

Available Boot Loaders on ESP:
          ESP: /boot/efi (/dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3)
         File: └─/EFI/systemd/systemd-bootx64.efi (systemd-boot 241)
         File: └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 241)

Boot Loaders Listed in EFI Variables:
        Title: UEFI OS
           ID: 0x0007
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3
         File: └─/EFI/BOOT/BOOTX64.EFI

        Title: UEFI OS
           ID: 0x0008
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/2c464620-9669-44c3-985f-f9a1505fab4d
         File: └─/EFI/BOOT/BOOTX64.EFI

Boot Loader Entries:
        $BOOT: /boot/efi (/dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3)

Default Boot Loader Entry:
        title: Proxmox Virtual Environment (5.13.19-6-pve)
           id: proxmox-5.13.19-6-pve.conf
       source: /boot/efi/loader/entries/proxmox-5.13.19-6-pve.conf
      version: 5.13.19-6-pve
        linux: /EFI/proxmox/5.13.19-6-pve/vmlinuz-5.13.19-6-pve
       initrd: /EFI/proxmox/5.13.19-6-pve/initrd.img-5.13.19-6-pve
      options: root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt

Next one is E89E-57D3.
Code:
root@pve:/# mount /dev/disk/by-uuid/E89E-57D3 /boot/efi/
root@pve:/# bootctl
System:
     Firmware: UEFI 2.70 (American Megatrends 5.17)
  Secure Boot: disabled
   Setup Mode: setup
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 241
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✗ Support for XBOOTLDR partition
               ✗ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
WARNING: The boot loader reports different ESP UUID then detected!
          ESP: /dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3
         File: └─/EFI/BOOT/BOOTX64.EFI

Random Seed:
 Passed to OS: no
 System Token: not set
       Exists: no

Available Boot Loaders on ESP:
          ESP: /boot/efi (/dev/disk/by-partuuid/2c464620-9669-44c3-985f-f9a1505fab4d)
         File: └─/EFI/systemd/systemd-bootx64.efi (systemd-boot 241)
         File: └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 241)

Boot Loaders Listed in EFI Variables:
        Title: UEFI OS
           ID: 0x0007
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3
         File: └─/EFI/BOOT/BOOTX64.EFI

        Title: UEFI OS
           ID: 0x0008
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/2c464620-9669-44c3-985f-f9a1505fab4d
         File: └─/EFI/BOOT/BOOTX64.EFI

Boot Loader Entries:
        $BOOT: /boot/efi (/dev/disk/by-partuuid/2c464620-9669-44c3-985f-f9a1505fab4d)

Default Boot Loader Entry:
        title: Proxmox Virtual Environment (5.13.19-6-pve)
           id: proxmox-5.13.19-6-pve.conf
       source: /boot/efi/loader/entries/proxmox-5.13.19-6-pve.conf
      version: 5.13.19-6-pve
        linux: /EFI/proxmox/5.13.19-6-pve/vmlinuz-5.13.19-6-pve
       initrd: /EFI/proxmox/5.13.19-6-pve/initrd.img-5.13.19-6-pve
      options: root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt

From what I can see in these, everything seems to point to using kernel 5.13, but the efi "grub" during boot is defaulting itself to 5.15.
 
* what's the output of `efibootmgr`?
* please try also running `bootctl set-default ''` , `bootctl set-oneshot ''` (to make sure the version is not pinned in a systemd efivar)
 
Hi,

could you try to run proxmox-boot-tool init /dev/disk/by-partuuid/502b28d0-b228-4d2b-a967-0ae6ce4e9ed3 and see if that fixes the issue for you?

Thank you!
 
Last edited:
Same here with UEFI and systemd-boot.

I tried proxmox-boot-tool kernel pin 5.13.19-6-pve and then proxmox-boot-tool refresh, but doesn't work.

Yesterday, i made a change in the /etc/kernel/cmdline in the same HV and then i run "proxmox-boot-tool refresh". After that the system boot up with 5.13.19-6-pve.
 
Same here with UEFI and systemd-boot.

I tried proxmox-boot-tool kernel pin 5.13.19-6-pve and then proxmox-boot-tool refresh, but doesn't work.

Yesterday, i made a change in the /etc/kernel/cmdline in the same HV and then i run "proxmox-boot-tool refresh". After that the system boot up with 5.13.19-6-pve.
I have check the history and this is what works for me:

proxmox-boot-tool kernel pin 5.13.19-6-pve
proxmox-boot-tool reinit
proxmox-boot-tool refresh
 
  • Like
Reactions: Stoiko Ivanov
type: Boot Loader Specification Type #1 (.conf)
title: Proxmox Virtual Environment (6.2.16-20-pve) (default) (selected)
id: proxmox-6.2.16-20-pve.conf
source: /boot/efi/loader/entries/proxmox-6.2.16-20-pve.conf
version: 6.2.16-20-pve
linux: /EFI/proxmox/6.2.16-20-pve/vmlinuz-6.2.16-20-pve
initrd: /EFI/proxmox/6.2.16-20-pve/initrd.img-6.2.16-20-pve
options: root=ZFS=rpool/ROOT/pve-1 boot=zfs


This is set as the default kernel in the proxmox-boot-tool kernel pin and also in the bootctl set-default and even so, it wants to load the new kernel 6.5.11-7, but the machine freezes completely!
 
This is set as the default kernel in the proxmox-boot-tool kernel pin and also in the bootctl set-default and even so, it wants to load the new kernel 6.5.11-7, but the machine freezes completely!
depends on when you set the box up originally - I'd recommend to run `proxmox-boot-tool reinit` - after creating appropriate backups of course!
systemd-boot in an older version (I think the one shipped with PVE 6.x) had an issue where the set-default would just not work....
 
I have yet another Dell T340 that does not boot with kernel 6.5 on PVE 8.1. While trying to pin kernel 6.2, the command seemed to succeed but the machine was still trying to boot with 6.5.

Using proxmox-boot-tool reinit complained about some non-existant command (sorry, was too late and didn't saved the whole error message). This server was updated from 6.3 -> 6.4 -> 7.4 -> 8.1 and at some point the package systemd-boot got either removed or not installed as dependency/requirement. Once installed, rerun proxmox-boot-tool reinit + proxmox-boot-tool refresh and the kernel got pinned properly.
 
  • Like
Reactions: julicravo

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!