Changing selected kernel for boot (systemd-boot) on headless PVE Host?

N

n1nj4888

Well-Known Member
Jan 13, 2019
162
22
58
44
Hi There,

I currently have a 2-node PVE 6 cluster (and 3rd node QDevice), with the 2 nodes operating headless with PVE installed as ZFS on EFI (systemd-boot). If I want to test the new PVE Linux 5.3 kernel on one of the nodes, I understand that I just need to do:

Code: 
apt update && apt install pve-kernel-5.3
pve-efiboot-tool refresh

However, if I need to rollback the kernel version, or ideally have the need to manually select the kernel version to be booted at the next reboot, I believe I need to edit the
Code: 
/boot/efi/loader/loader.conf
to change the default boot kernel and then run
Code: 
pve-efiboot-tool refresh
to enable, and reboot to the new selected kernel.

On checking,
Code: 
/boot/efi/loader/loader.conf
does not exist on the PVE host, I suspect because of the following line in https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_systemd_boot

"The ESPs are not kept mounted during regular operation, in contrast to grub, which keeps an ESP mounted on /boot/efi. This helps to prevent filesystem corruption to the vfat formatted ESPs in case of a system crash, and removes the need to manually adapt /etc/fstab in case the primary boot device fails."

Therefore, what is the best way to be able to get change the default kernel that is default booted on a headless systemd-boot PVE node?

Thanks!
 
the most straight-forward way is to just remove the 5.3 kernel again ;)
 
Thanks @fabian

Ok, I assume the way to do that would simply be the following command, reboot and it would automatically boot the last installed kernel version?

Code: 
apt remove pve-kernel-5.3
pve-efiboot-tool refresh

I haven’t installed 5.3 yet, I just want to be confident I can revert easily if required.
 
just removing the pve-kernel-5.3.x-y-pve packages should be enough (it will also remove the meta-package pve-kernel-5.3, and then automatically regenerate the boot loader entries for the remaining 5.0 kernels)
 
  • Like
Reactions: muyue
@fabian,

So I thought I'd test out and upgrade to/rollback from Kernel 5.3 on a PVE6 VM before I looked at upgrading the Physical PVE Node and, even though the 5.3 kernel packages (pve-kernel-5.3 pve-kernel-5.3.7-1-pve) were listed as removed, the VM still reboots back into Kernel 5.3?

Here's some additional information on what I did:

Install PVE 6.0.1 fresh as a VM (ZFS on EFI)
Add no-subscription repo and do full update to latest via the Web GUI
Reboot
Add pvetest repo
Execute in the VM console:
Code: 
apt update
Execute in the VM console:
Code: 
apt install pve-kernel-5.3
Reboot
VM reboots to kernel 5.3
Code: 
# uname -a
Linux pve6-test1 5.3.7-1-pve #1 SMP PVE 5.3.7-1 (Wed, 23 Oct 2019 19:00:21 +0200) x86_64 GNU/Linux

Now, when testing a rollback of the 5.3 kernel, I did the following:

Code: 
# apt remove pve-kernel-5.3.7-1-pve
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  pve-kernel-5.3 pve-kernel-5.3.7-1-pve
0 upgraded, 0 newly installed, 2 to remove and 19 not upgraded.
After this operation, 284 MB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 56342 files and directories currently installed.)
Removing pve-kernel-5.3 (6.0-11) ...
Removing pve-kernel-5.3.7-1-pve (5.3.7-1) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
update-initramfs: Deleting /boot/initrd.img-5.3.7-1-pve
run-parts: executing /etc/kernel/postrm.d/pve-auto-removal 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-pve-efiboot 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
Re-executing '/etc/kernel/postrm.d/zz-pve-efiboot' in new private mount namespace..
mount: /var/tmp/espmounts/5C7B-FFED: wrong fs type, bad option, bad superblock on /dev/sda2, missing codepage or helper program, or other error.
mount of /dev/disk/by-uuid/5C7B-FFED failed - skipping
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.0.21-3-pve
Found initrd image: /boot/initrd.img-5.0.21-3-pve
Found linux image: /boot/vmlinuz-5.0.15-1-pve
Found initrd image: /boot/initrd.img-5.0.15-1-pve
Found memtest86+ image: /ROOT/pve-1@/boot/memtest86+.bin
Found memtest86+ multiboot image: /ROOT/pve-1@/boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done

You'll notice the mount: /var/tmp/espmounts/5C7B-FFED error above...

The strange thing is, even though pve-kernel-5.3 has apparently been removed, when the VM is rebooted, the default boot screen option is still listed as Kernel 5.3.x and the VM still boots into the 5.3 Kernel:

Code: 
root@pve6-test1:~# uname -a
Linux pve6-test1 5.3.7-1-pve #1 SMP PVE 5.3.7-1 (Wed, 23 Oct 2019 19:00:21 +0200) x86_64 GNU/Linux
 
well, what FS is on /dev/sda2?
 
Looks like /dev/sda2 is the EFI System as per the following fdisk -l output from the VM on which this test PVE6 is installed.

1572851343562.png

Similar output is returned by fdisk -l on the actual physical PVE6 host hosting this test PVE6 VM... When I installed the VM, I just selected the ZFS (Raid 0) option and haven't amended the partition layout/filesystems since the initial install...

Thanks!
 
can you try mounting it to some temporary path? since you are booting the 5.3 kernel, it must have worked previously?
 
fabian said:
can you try mounting it to some temporary path? since you are booting the 5.3 kernel, it must have worked previously?
Click to expand...

Happy to test this out, especially since it's a VM but (excuse my ignorance:) ), could you please give me the list of commands that I should run to accomplish this?

Thanks!
 
Code: 
mkdir /tmp/test
mount /dev/sda2 /tmp/test
umount /tmp/test
 
what do lsmod nls_iso88591-1 and uname -a say?
 
Code: 
lsmod nls_iso88591-1
command just errors with "Usage: lsmod" (I think there is a mistake in your commands above?)

Code: 
lsmod
(run jus as "lsmod") output as follows:

1572859400032.png

Code: 
uname -a
output as follows:

1572859575238.png

Thanks!
 
yes, sorry. could you just try lsmod | grep nls (or alternatively, modinfo nls_iso8859_1?
 
Results for the two commands below - Note,
Code: 
lsmod | grep nls
does not return anything ...

1572860663823.png
 
Hi @fabian

I see you’ve logged a bug report for this so I’ll also monitor the updates there.

https://bugzilla.proxmox.com/show_bug.cgi?id=2448

Is there anyway currently to work around this issue prior to it occurring (ie before removing the currently booted kernel) or to fix the module nls_iso8859_1 loading once already in this situation?

thanks for all your help with this!
 
yes! sorry, it seems I missed actually posting the following reply that I wrote in parallel to the bug report :-/

thanks for your report! I can reproduce the issue, seems like you found a bug with our pve-kernel-helper and removing the currently running kernel under certain circumstances.. I filed https://bugzilla.proxmox.com/show_bug.cgi?id=2448 for tracking.

an immediate workaround is to run the following sequence:
Code: 
apt install pve-kernel-5.3
apt remove "pve-kernel-5.3-*"

the first step will make the missing module file available again (and load it, since it triggers a "pve-efiboot-tool refresh"). the second can then successfully remove the kernel also from the ESP. the issue only occurs if
- the currently running kernel gets removed
- no call to 'pve-efiboot-tool refresh' has happened since rebooting the system
 
Hi @fabian

I tried the work around above and the following is the output. On rebooting the 5.3 Kernel is still shown in the EFI Boot Menu (as default) and is loaded despite the output from the commands above?

Thanks!

Code: 
root@pve6-test1:~# uname -a
Linux pve6-test1 5.3.7-1-pve #1 SMP PVE 5.3.7-1 (Wed, 23 Oct 2019 19:00:21 +0200) x86_64 GNU/Linux
root@pve6-test1:~# apt install pve-kernel-5.3
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  pve-kernel-5.3.7-1-pve
The following NEW packages will be installed:
  pve-kernel-5.3 pve-kernel-5.3.7-1-pve
0 upgraded, 2 newly installed, 0 to remove and 19 not upgraded.
Need to get 59.5 MB of archives.
After this operation, 284 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://download.proxmox.com/debian/pve buster/pvetest amd64 pve-kernel-5.3.7-1-pve amd64 5.3.7-1 [59.5 MB]
Get:2 http://download.proxmox.com/debian/pve buster/pvetest amd64 pve-kernel-5.3 all 6.0-11 [3,004 B]
Fetched 59.5 MB in 34s (1,749 kB/s)
Selecting previously unselected package pve-kernel-5.3.7-1-pve.
(Reading database ... 49788 files and directories currently installed.)
Preparing to unpack .../pve-kernel-5.3.7-1-pve_5.3.7-1_amd64.deb ...
Unpacking pve-kernel-5.3.7-1-pve (5.3.7-1) ...
Selecting previously unselected package pve-kernel-5.3.
Preparing to unpack .../pve-kernel-5.3_6.0-11_all.deb ...
Unpacking pve-kernel-5.3 (6.0-11) ...
Setting up pve-kernel-5.3.7-1-pve (5.3.7-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
update-initramfs: Generating /boot/initrd.img-5.3.7-1-pve
run-parts: executing /etc/kernel/postinst.d/pve-auto-removal 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-pve-efiboot 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/5C7B-FFED
        Copying kernel and creating boot-entry for 5.0.21-3-pve
        Copying kernel and creating boot-entry for 5.3.7-1-pve
        Removing old version 5.0.15-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.3.7-1-pve
Found initrd image: /boot/initrd.img-5.3.7-1-pve
Found linux image: /boot/vmlinuz-5.0.21-3-pve
Found initrd image: /boot/initrd.img-5.0.21-3-pve
Found linux image: /boot/vmlinuz-5.0.15-1-pve
Found initrd image: /boot/initrd.img-5.0.15-1-pve
Found memtest86+ image: /ROOT/pve-1@/boot/memtest86+.bin
Found memtest86+ multiboot image: /ROOT/pve-1@/boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done
Setting up pve-kernel-5.3 (6.0-11) ...
root@pve6-test1:~# apt remove "pve-kernel-5.3-*"
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'pve-kernel-5.3' for regex 'pve-kernel-5.3-*'
Note, selecting 'pve-kernel-5.3.1-1-pve' for regex 'pve-kernel-5.3-*'
Note, selecting 'pve-kernel-5.3.7-1-pve' for regex 'pve-kernel-5.3-*'
Package 'pve-kernel-5.3.1-1-pve' is not installed, so not removed
The following packages will be REMOVED:
  pve-kernel-5.3 pve-kernel-5.3.7-1-pve
0 upgraded, 0 newly installed, 2 to remove and 19 not upgraded.
After this operation, 284 MB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 56342 files and directories currently installed.)
Removing pve-kernel-5.3 (6.0-11) ...
Removing pve-kernel-5.3.7-1-pve (5.3.7-1) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
update-initramfs: Deleting /boot/initrd.img-5.3.7-1-pve
run-parts: executing /etc/kernel/postrm.d/pve-auto-removal 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-pve-efiboot 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
Re-executing '/etc/kernel/postrm.d/zz-pve-efiboot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/5C7B-FFED
        Copying kernel and creating boot-entry for 5.0.15-1-pve
        Copying kernel and creating boot-entry for 5.0.21-3-pve
No linux-image /boot/vmlinuz-5.3.7-1-pve found - skipping
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 5.3.7-1-pve /boot/vmlinuz-5.3.7-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.0.21-3-pve
Found initrd image: /boot/initrd.img-5.0.21-3-pve
Found linux image: /boot/vmlinuz-5.0.15-1-pve
Found initrd image: /boot/initrd.img-5.0.15-1-pve
Found memtest86+ image: /ROOT/pve-1@/boot/memtest86+.bin
Found memtest86+ multiboot image: /ROOT/pve-1@/boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done
root@pve6-test1:~# uname -a
Linux pve6-test1 5.3.7-1-pve #1 SMP PVE 5.3.7-1 (Wed, 23 Oct 2019 19:00:21 +0200) x86_64 GNU/Linux
root@pve6-test1:~# reboot

Following the reboot, the 5.3 Kernel is still shown as the default EFI boot menu option and is loaded as before:

Code: 
root@pve6-test1:~# uname -a
Linux pve6-test1 5.3.7-1-pve #1 SMP PVE 5.3.7-1 (Wed, 23 Oct 2019 19:00:21 +0200) x86_64 GNU/Linux

Thanks!
 
n1nj4888 said:
On rebooting the 5.3 Kernel is still shown in the EFI Boot Menu (as default) and is loaded despite the output from the commands above?
Click to expand...

The reason for that is that the kernel is still one the EFI boot partition, I mean the part where our script tried to clean that kernel and the initrd there up failed, so :)

n1nj4888 said:
Following the reboot, the 5.3 Kernel is still shown as the default EFI boot menu option and is loaded as before:
Click to expand...

OK, that now seems like a different issue.
n1nj4888 said:
No linux-image /boot/vmlinuz-5.3.7-1-pve found - skipping
Click to expand...

Hmm, why does the refresh step still checks that kernel? What's the output of pve-efiboot-tool kernel list

If it is included there can you try to:
Code: 
pve-efiboot-tool kernel remove 5.3.7-1-pve
pve-efiboot-tool refresh
 
The following is the output of pve-efiboot-tool kernel list

Code: 
root@pve6-test1:~# pve-efiboot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
5.0.15-1-pve
5.0.21-3-pve
5.3.7-1-pve

So, I then ran the following commands as you stated above:

Code: 
root@pve6-test1:~# pve-efiboot-tool kernel remove 5.3.7-1-pve
Kernel '5.3.7-1-pve' not found in manual kernel list.
root@pve6-test1:~# pve-efiboot-tool refresh
Running hook script 'pve-auto-removal'..
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
mount: /var/tmp/espmounts/5C7B-FFED: wrong fs type, bad option, bad superblock on /dev/sda2, missing codepage or helper program, or other error.
mount of /dev/disk/by-uuid/5C7B-FFED failed - skipping
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back