Unable to remove kernel

The steps are in this post:

Do the first 3 steps for each UUID in /etc/kernel/proxmox-boot-uuids.
Hello thank you very much, I performed everything and I created a small guide or memorandum in case it should happen on other nodes, so I would like to know what you think.

Code:
###############################################
# CORRECT SEQUENCE FOR ESP CLEANUP + REMOVAL OF 6.x KERNELS
###############################################

echo "=== 1) Identify ESP ==="
cat /etc/kernel/proxmox-boot-uuids
ls -al /dev/disk/by-uuid/B130-7F39   # -> ../../sda2

echo "=== 2) Mount ESP ==="
mkdir -p /tmp/myesp
mount /dev/sda2 /tmp/myesp

echo "=== 3) List kernels present on the ESP ==="
ls -l /tmp/myesp

echo "=== 4) Delete ALL old kernels (6.x series) ==="
rm -f /tmp/myesp/initrd.img-6*
rm -f /tmp/myesp/vmlinuz-6*

echo "=== 5) Verify that NO 6.x files remain ==="
ls -l /tmp/myesp

echo "=== 6) Check ESP free space ==="
df -h /tmp/myesp
du -ah /tmp/myesp

echo "=== 7) Unmount ESP ==="
umount /tmp/myesp

echo "=== 8) Repair dpkg if necessary ==="
apt-get -f install

echo "=== 9) Remove 6.x kernels ==="
apt purge proxmox-kernel-6.*

echo "=== 10) Verify installed kernels ==="
dpkg -l | grep proxmox-kernel

echo "=== 11) Verify boot configuration ==="
proxmox-boot-tool status

echo "=== COMPLETED ==="

run-parts: executing /etc/kernel/postrm.d/zz-update-grub 6.14.11-2-pve /boot/vmlinuz-6.14.11-2-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-4-pve
Found initrd image: /boot/initrd.img-7.0.14-4-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found linux image: /boot/vmlinuz-7.0.6-2-pve
Found initrd image: /boot/initrd.img-7.0.6-2-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
rmdir: failed to remove '/lib/modules/6.14.11-2-pve': Directory not empty
dpkg: warning: while removing proxmox-kernel-6.14.11-2-pve-signed, directory '/lib/modules/6.14.11-2-pve' not empty so not removed
Purging configuration files for proxmox-kernel-6.14 (6.14.11-9) ...
Purging configuration files for proxmox-kernel-6.8 (6.8.12-15) ...
root@pvetest:~# dpkg -l | grep proxmox-kernel
ii proxmox-kernel-7.0 7.0.14-4 amd64 Latest Proxmox Kernel Image
ii proxmox-kernel-7.0.14-4-pve-signed 7.0.14-4 amd64 Proxmox Kernel Image (signed)
ii proxmox-kernel-7.0.6-2-pve-signed 7.0.6-2 amd64 Proxmox Kernel Image (signed)
ii proxmox-kernel-helper 9.2.0 all Function for various kernel maintenance tasks.
root@pvetest:~#
root@pvetest:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
B130-7F39 is configured with: grub (versions: 7.0.14-4-pve, 7.0.6-2-pve)
root@pvetest:~#


Code:
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 6.14.11-2-pve /boot/vmlinuz-6.14.11-2-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-4-pve
Found initrd image: /boot/initrd.img-7.0.14-4-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found linux image: /boot/vmlinuz-7.0.6-2-pve
Found initrd image: /boot/initrd.img-7.0.6-2-pve
/usr/sbin/grub-probe: error: unknown filesystem.
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
rmdir: failed to remove '/lib/modules/6.14.11-2-pve': Directory not empty
dpkg: warning: while removing proxmox-kernel-6.14.11-2-pve-signed, directory '/lib/modules/6.14.11-2-pve' not empty so not removed
Purging configuration files for proxmox-kernel-6.14 (6.14.11-9) ...
Purging configuration files for proxmox-kernel-6.8 (6.8.12-15) ...
root@pvetest:~# dpkg -l | grep proxmox-kernel
ii  proxmox-kernel-7.0                   7.0.14-4                             amd64        Latest Proxmox Kernel Image
ii  proxmox-kernel-7.0.14-4-pve-signed   7.0.14-4                             amd64        Proxmox Kernel Image (signed)
ii  proxmox-kernel-7.0.6-2-pve-signed    7.0.6-2                              amd64        Proxmox Kernel Image (signed)
ii  proxmox-kernel-helper                9.2.0                                all          Function for various kernel maintenance tasks.
root@pvetest:~#
root@pvetest:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
B130-7F39 is configured with: grub (versions: 7.0.14-4-pve, 7.0.6-2-pve)
root@pvetest:~#
 
Good to read that you have resolved it.

1 & 2 can be combined: mount /dev/disk/by-uuid/B130-7F39 /tmp/myesp but otherwise, yes those are the steps and checks.

I see that you are booting legacy bios on this node. Keep in mind that when you are booting UEFI (on other nodes) the layout of your ESP will be different.
 
Good to read that you have resolved it.

1 & 2 can be combined: mount /dev/disk/by-uuid/B130-7F39 /tmp/myesp but otherwise, yes those are the steps and checks.

I see that you are booting legacy bios on this node. Keep in mind that when you are booting UEFI (on other nodes) the layout of your ESP will be different.

Here’s what I noticed the first time I ran into the ESP‑full issue and went through the cleanup procedure: when I mounted the ESP on the production nodes, the kernel files were not located in the root of the partition. The layout was slightly different compared to the node that boots in legacy BIOS.


From what I understand, though, regardless of whether the node is booting in legacy BIOS or UEFI, the kernels that need to be removed (and then purged) are always the old 6.x ones — they just happen to be stored in different paths depending on the boot mode.
 
I wanted to know what the development team thinks of this problem, maybe put a patch or script to avoid blockages during the upgrade. That would be a great thing.