I made an account just to thank fabian. 
His post was very helpful even though the OP is going to do a reinstall. Following the logic of the advice that fabian gave in the thread https://forum.proxmox.com/threads/post-pve-9-0x-upgrade-from-pve-8.171375/#post-799330 was helpful though the issue there was not my exact issue.
Hopefully this helps others as well.
TLDR: The kernel postinst scripts do a mount you don't see with df until you recreate it. You have to mount the filesystem for EFI that is noted by the uuid in /var/tmp/espmounts error to clean up the EFI boot area since it is 512MB and data from 3 kernels (6.14.8-2-pve, 6.14.11-1-pve and 6.14.11-2-pve) is too much for it to handle.
The issue here for me was not initially seeing the space on the filesystem that holds the EFI folder where the kernels store their files since it was not mounted.
Once mounted I clearly saw I had no free space on filesystem the /var/tmp/espmounts/2CF8-4B37/EFI folder as stated by the error "cp: error writing '/var/tmp/espmounts/2CF8-4B37/EFI/proxmox/6.14.11-2-pve/initrd.img-6.14.11-2-pve': No space left on device".
To get apt purge, apt dist-upgrade, or apt autoremove to complete I had to mount /dev/disk/by-uuid/2CF8-4B37 for df -h to show me the actual used space on the filesystem the error was referring to since doing df -h /var/tmp/espmounts/2CF8-4B37 shows my rpool/ROOT/pve-1 free space.
After mounting the device by uuid to a new mount point (/tmp/cleanitup) I could see df -h was now showing 511MB of space with 511MB used.
Looking at uname -r showed my what kernel was running (6.14.11-1) allowing me to rm any files related to other kernel versions to free space so I could get apt dist-update to complete (as well as autoremove and purge). Use caution when deleting files since you can end up with a non bootable system.
The error I saw during apt dist-upgrade was the same as apt autoremove seen below.
After removing 6.14.8-2-pve via the command below I had enough space:
So I then ran the code below that finally worked and I was able to reboot.
Caveats: This isn't advice it is just what worked for me. Have a backup. This is all on my non-production homelab using zfs on all drives doing gpu passthrough on multiple gpus. Running amdgpu-install with dkms and prior to that testing out nvidia cards on the host to learn about ai infrastructure while between jobs may or may not have contributed to my EFI space issues via some residual files or some other issue but you can gain experience though pain (yours or someone else's) and learning how lxc gpu passthough via amd and nvidia works is worth it ;-).
Hope this helps someone.
Thanks again to fabian.
Now on to getting apparmor in 9.x.x to stop killing my samba shares mid file transfer...
His post was very helpful even though the OP is going to do a reinstall. Following the logic of the advice that fabian gave in the thread https://forum.proxmox.com/threads/post-pve-9-0x-upgrade-from-pve-8.171375/#post-799330 was helpful though the issue there was not my exact issue.Hopefully this helps others as well.
TLDR: The kernel postinst scripts do a mount you don't see with df until you recreate it. You have to mount the filesystem for EFI that is noted by the uuid in /var/tmp/espmounts error to clean up the EFI boot area since it is 512MB and data from 3 kernels (6.14.8-2-pve, 6.14.11-1-pve and 6.14.11-2-pve) is too much for it to handle.
The issue here for me was not initially seeing the space on the filesystem that holds the EFI folder where the kernels store their files since it was not mounted.
Once mounted I clearly saw I had no free space on filesystem the /var/tmp/espmounts/2CF8-4B37/EFI folder as stated by the error "cp: error writing '/var/tmp/espmounts/2CF8-4B37/EFI/proxmox/6.14.11-2-pve/initrd.img-6.14.11-2-pve': No space left on device".
To get apt purge, apt dist-upgrade, or apt autoremove to complete I had to mount /dev/disk/by-uuid/2CF8-4B37 for df -h to show me the actual used space on the filesystem the error was referring to since doing df -h /var/tmp/espmounts/2CF8-4B37 shows my rpool/ROOT/pve-1 free space.
After mounting the device by uuid to a new mount point (/tmp/cleanitup) I could see df -h was now showing 511MB of space with 511MB used.
Code:
mkdir /tmp/cleanitup
mount /dev/disk/by-uuid/2CF8-4B37 /tmp/cleanitup/
df -h /tmp/cleanitup
find /tmp/cleanitup/
uname -rLooking at uname -r showed my what kernel was running (6.14.11-1) allowing me to rm any files related to other kernel versions to free space so I could get apt dist-update to complete (as well as autoremove and purge). Use caution when deleting files since you can end up with a non bootable system.
The error I saw during apt dist-upgrade was the same as apt autoremove seen below.
Code:
#apt autoremove
Summary:
Upgrading: 0, Installing: 0, Removing: 0, Not Upgrading: 0
2 not fully installed or removed.
Space needed: 0 B / 197 GB available
Setting up proxmox-kernel-6.14.11-2-pve-signed (6.14.11-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/dkms 6.14.11-2-pve /boot/vmlinuz-6.14.11-2-pve
Automatic installation of modules for kernel 6.14.11-2-pve was skipped since the kernel headers for this kernel do not seem to be installed.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.14.11-2-pve /boot/vmlinuz-6.14.11-2-pve
update-initramfs: Generating /boot/initrd.img-6.14.11-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/2CF8-4B37
Copying kernel and creating boot-entry for 6.14.11-1-pve
Copying kernel and creating boot-entry for 6.14.11-2-pve
cp: error writing '/var/tmp/espmounts/2CF8-4B37/EFI/proxmox/6.14.11-2-pve/initrd.img-6.14.11-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.14.11-2-pve-signed.postinst line 20.
dpkg: error processing package proxmox-kernel-6.14.11-2-pve-signed (--configure):
installed proxmox-kernel-6.14.11-2-pve-signed package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-kernel-6.14:
proxmox-kernel-6.14 depends on proxmox-kernel-6.14.11-2-pve-signed | proxmox-kernel-6.14.11-2-pve; however:
Package proxmox-kernel-6.14.11-2-pve-signed is not configured yet.
Package proxmox-kernel-6.14.11-2-pve is not installed.
Package proxmox-kernel-6.14.11-2-pve-signed which provides proxmox-kernel-6.14.11-2-pve is not configured yet.
dpkg: error processing package proxmox-kernel-6.14 (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
proxmox-kernel-6.14.11-2-pve-signed
proxmox-kernel-6.14
Error: Sub-process /usr/bin/dpkg returned an error code (1)After removing 6.14.8-2-pve via the command below I had enough space:
Code:
rm -rf /tmp/cleanitup/EFI/proxmox/6.14.8-2-pve
df -h /tmp/cleanitup/EFI/
Filesystem Size Used Avail Use% Mounted on
/dev/sdd2 511M 333M 179M 66% /tmp/cleanitupSo I then ran the code below that finally worked and I was able to reboot.
Code:
apt autoremove
apt update
apt dist-upgradeCaveats: This isn't advice it is just what worked for me. Have a backup. This is all on my non-production homelab using zfs on all drives doing gpu passthrough on multiple gpus. Running amdgpu-install with dkms and prior to that testing out nvidia cards on the host to learn about ai infrastructure while between jobs may or may not have contributed to my EFI space issues via some residual files or some other issue but you can gain experience though pain (yours or someone else's) and learning how lxc gpu passthough via amd and nvidia works is worth it ;-).
Hope this helps someone.
Thanks again to fabian.
Now on to getting apparmor in 9.x.x to stop killing my samba shares mid file transfer...
Last edited: