apt autoremove keeps too many kernels - out of ESP space

ahovda

Member
Sep 2, 2019
24
11
23
46
I found this portion in /usr/share/pve-kernel-helper/scripts/functions

Code:
# Mark as not-for-autoremoval those kernel packages that are:
#  - the currently booted version, if still installed
#  - the kernel version we've been called for
#  - the latest kernel version (as determined by debian version number)
#  - the second-latest kernel version
#  - the latest kernel version of each series (e.g. 4.13, 4.15, 5.0) by
#    marking the meta-packages
#  - the currently pinned kernel if any

However, I'm using systemd-boot and the default EFI partition is 512MB on most of the machines in my clusters, so when I perform an upgrade, it sometimes fails because the ESP runs out of space when apt tries to keep four or more kernels on there. Or it would have succeeded but fails because the new kernel is installed before the old is cleaned up.

Yes, I did manually convert from grub to systemd-boot using proxmox-boot-tool for ....reasons. And I found that if you put rootfs directly on ZFS the installer defaults to 1GB ESP using systemd-boot and the problem would not be there, but these systems have been upgraded from PVE 5-6-7 already and it would be a hassle to reinstall them all, or have to revert to grub...

It would be great if we could shortcut this logic somewhat. For myself, it would be sufficient to keep
  • The currently running kernel
  • The latest kernel from the metapackage required by the proxmox-ve package
If the reboot fails I will have one known good kernel to recover into. I found I could hack it by reducing this list in /usr/share/pve-kernel-helper/scripts/functions:
Code:
        kernels="$(cat <<-EOF
                $running_version
                $install_version
                $manual_kernels
                $latest_2_versions
                $series_metapackages
                $oldseries_latest_kernel
                $pinned_kernel
                $nextboot_kernel
                EOF
        )"

down to
Code:
        kernels="$(cat <<-EOF
                $running_version
                $install_version
                $manual_kernels
                $pinned_kernel
                $nextboot_kernel
                EOF
        )"

which does what I described (maybe?)

And I believe it is okay to remove $series_metapackages in this case too since proxmox-ve depends on the last metapackage anyway?
 
Last edited:
Yes, I did manually convert from grub to systemd-boot using proxmox-boot-tool for ....reasons.
I think this might be the culprit - the old grub install might still be occupying space there - if you're sure that UEFI boot with systemd-boot works - it might be easiest to just reformat, reinit all esps and run a clean afterwards to get rid of the old UUIDs
In any case - make sure you have access to the machine in case something does not work out and a working and tested backup!

And I found that if you put rootfs directly on ZFS the installer defaults to 1GB ESP using systemd-boot and the problem would not be there
This should not be the case for the Proxmox VE ISO - the ESP is still (as it's been since afair 5.4) created with 512MB size

I hope this helps!
 
the old grub install might still be occupying space there
Good point, but I've completely purged all grub packages from the system and cleaned ESP, so there are just a few files on them now.
Code:
root@osl101pve:~# mount /dev/sda2 /boot/efi/
root@osl101pve:~# find /boot/ -ls
        2      9 drwxr-xr-x   4 root     root           12 Nov 24 21:11 /boot/
        1      4 drwxr-xr-x   4 root     root         4096 Jan  1  1970 /boot/efi
     1533      4 drwxr-xr-x   6 root     root         4096 Dec  2 10:53 /boot/efi/EFI
     1538      4 drwxr-xr-x   4 root     root         4096 Nov 24 21:09 /boot/efi/EFI/proxmox
     1541      4 drwxr-xr-x   2 root     root         4096 Jun 23 12:51 /boot/efi/EFI/proxmox/5.15.35-3-pve
     1544  10616 -rwxr-xr-x   1 root     root     10867488 Jun 17 13:42 /boot/efi/EFI/proxmox/5.15.35-3-pve/vmlinuz-5.15.35-3-pve
     1545  61344 -rwxr-xr-x   1 root     root     62814759 Jun 23 12:51 /boot/efi/EFI/proxmox/5.15.35-3-pve/initrd.img-5.15.35-3-pve
     1546      4 drwxr-xr-x   2 root     root         4096 Nov 24 21:09 /boot/efi/EFI/proxmox/5.15.74-1-pve
     1549  11044 -rwxr-xr-x   1 root     root     11308736 Nov 14 20:17 /boot/efi/EFI/proxmox/5.15.74-1-pve/vmlinuz-5.15.74-1-pve
     1550  59684 -rwxr-xr-x   1 root     root     61113402 Nov 24 21:11 /boot/efi/EFI/proxmox/5.15.74-1-pve/initrd.img-5.15.74-1-pve
     1551      4 drwxr-xr-x   2 root     root         4096 Nov 24 21:09 /boot/efi/EFI/systemd
     1553     92 -rwxr-xr-x   1 root     root        93019 Aug  7 15:25 /boot/efi/EFI/systemd/systemd-bootx64.efi
     1554      4 drwxr-xr-x   2 root     root         4096 Nov 24 21:09 /boot/efi/EFI/BOOT
     1556     92 -rwxr-xr-x   1 root     root        93019 Aug  7 15:25 /boot/efi/EFI/BOOT/BOOTX64.EFI
     1557      4 drwxr-xr-x   2 root     root         4096 Jun 13 22:12 /boot/efi/EFI/Linux
     1558      4 drwxr-xr-x   3 root     root         4096 Nov 24 21:11 /boot/efi/loader
     1562      4 drwxr-xr-x   2 root     root         4096 Nov 24 21:09 /boot/efi/loader/entries
     1565      4 -rwxr-xr-x   1 root     root          223 Nov 24 21:11 /boot/efi/loader/entries/proxmox-5.15.35-3-pve.conf
     1566      4 -rwxr-xr-x   1 root     root          223 Nov 24 21:11 /boot/efi/loader/entries/proxmox-5.15.74-1-pve.conf
     1567      4 -rwxr-xr-x   1 root     root           28 Nov 24 21:11 /boot/efi/loader/loader.conf
     1568      4 -rwxr-xr-x   1 root     root          512 Nov 24 21:09 /boot/efi/loader/random-seed
    40396   2077 -rw-r--r--   1 root     root      6082215 Nov 14 20:17 /boot/System.map-5.15.74-1-pve
    39508  10957 -rw-r--r--   1 root     root     11308736 Nov 14 20:17 /boot/vmlinuz-5.15.74-1-pve
    39507    113 -rw-r--r--   1 root     root       261134 Nov 14 20:17 /boot/config-5.15.74-1-pve
   189785  61321 -rw-r--r--   1 root     root     62814759 Jun 23 12:51 /boot/initrd.img-5.15.35-3-pve
   177773  10541 -rw-r--r--   1 root     root     10867488 Jun 17 13:42 /boot/vmlinuz-5.15.35-3-pve
   177772    113 -rw-r--r--   1 root     root       260941 Jun 17 13:42 /boot/config-5.15.35-3-pve
    64403  59725 -rw-r--r--   1 root     root     61113402 Nov 24 21:11 /boot/initrd.img-5.15.74-1-pve
    51739      9 drwxr-xr-x   2 root     root            6 Nov 24 21:10 /boot/pve
    55885      1 lrwxrwxrwx   1 root     root           30 Nov 24 21:09 /boot/pve/initrd.img-5.15 -> /boot/initrd.img-5.15.74-1-pve
    64373      1 lrwxrwxrwx   1 root     root           22 Nov 24 21:10 /boot/pve/vmlinuz -> /boot/pve/vmlinuz-5.15
    41711      1 lrwxrwxrwx   1 root     root           27 Nov 24 21:09 /boot/pve/vmlinuz-5.15 -> /boot/vmlinuz-5.15.74-1-pve
    64374      1 lrwxrwxrwx   1 root     root           25 Nov 24 21:10 /boot/pve/initrd.img -> /boot/pve/initrd.img-5.15
   177771   2077 -rw-r--r--   1 root     root      6080219 Jun 17 13:42 /boot/System.map-5.15.35-3-pve

This should not be the case for the Proxmox VE ISO - the ESP is still (as it's been since afair 5.4) created with 512MB size
You're absolutely right, I just installed the latest PVE on zfs in a VM to check for myself and the ESPs are indeed 512MB.

1669976915669.png

Code:
root@pvetestzfsboot:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
F933-06AE is configured with: uefi (versions: 5.15.74-1-pve)
F933-654F is configured with: uefi (versions: 5.15.74-1-pve)

root@pvetestzfsboot:~# bootctl
Couldn't find EFI system partition. It is recommended to mount it to /boot or /efi.
Alternatively, use --esp-path= to specify path to mount point.
System:
     Firmware: UEFI 2.70 (EDK II 1.00)
  Secure Boot: disabled
   Setup Mode: setup
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 247.3-7+deb11u1
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
WARNING: The boot loader reports different ESP UUID then detected!
          ESP: /dev/disk/by-partuuid/4b507f9b-9bf3-44dc-9ac8-ac374ea87d7c
         File: └─/EFI/systemd/systemd-bootx64.efi

Random Seed:
 Passed to OS: no
 System Token: not set

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x000A
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/4b507f9b-9bf3-44dc-9ac8-ac374ea87d7c
         File: └─/EFI/systemd/systemd-bootx64.efi

        Title: Linux Boot Manager
           ID: 0x0009
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/7bb73b4f-848b-4814-95a9-35ca3b9e4d0e
         File: └─/EFI/systemd/systemd-bootx64.efi
So in that case anyone with root on zfs will run into disk space issues on the ESPs?
 
The ESPs are not mounted by default - are you sure that the contents at /boot/efi are from one of the ESPs mentioned in /etc/kernel/proxmox-boot-uuids?
 
The ESPs are not mounted by default - are you sure that the contents at /boot/efi are from one of the ESPs mentioned in /etc/kernel/proxmox-boot-uuids?
Sure, I mounted it first as shown in my post, just to list the contents here.
 
Sure, I mounted it first as shown in my post, just to list the contents here.
sorry - I overlooked that - from a quick glance - I see 2 kernelimages+initrds (each roughly 70M) on /boot/efi ? - so that should work nicely with 512M ESP?
 
I checked again and sorry for the confusion. I think some kernel packages did not autoremove since they may have been installed manually and I did not run apt-mark auto on them after. Afaict there are still no more than 4 kernels on the ESP, which fit nicely on a 512MB partition, even if there may be more pve-kernel-* packages installed on the system.

I found that the systems I have trouble with have 256MB ESPs, but these are older machines (actually new chassis/hw but migrated/upgraded OS) that were converted from BIOS to UEFI boot post-install. Seems to be time for a reinstall of those machines then... Thanks again for you help.
 
I found that the systems I have trouble with have 256MB ESPs, but these are older machines (actually new chassis/hw but migrated/upgraded OS) that were converted from BIOS to UEFI boot post-install. Seems to be time for a reinstall of those machines then... Thanks again for you help.
hmm - yes - that's sadly an issue where I can't think of a simple workaround (should you run into this once again - you could manually partition the new OS drives (with a larger ESP) and sync partition for partion ) - but it might not be the worst idea to get a fresh setup (also shows quite well what modifications you did and still need in my experience)

Glad we figured out what was causing it :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!