apt-get dist-upgrade failed on VE 7.4-17

[toolix]

My PVE 7.4-17 shows the following updates

apt list --upgradable

caddy/any-version 2.7.6 amd64 [upgradable from: 2.7.5]
curl/oldstable-security,oldstable-security 7.74.0-1.3+deb11u11 amd64 [upgradable from: 7.74.0-1.3+deb11u10]
intel-microcode/oldstable-security,oldstable-security 3.20231114.1~deb11u1 amd64 [upgradable from: 3.20230808.1~deb11u1]
libcurl3-gnutls/oldstable-security,oldstable-security 7.74.0-1.3+deb11u11 amd64 [upgradable from: 7.74.0-1.3+deb11u10]
libcurl4/oldstable-security,oldstable-security 7.74.0-1.3+deb11u11 amd64 [upgradable from: 7.74.0-1.3+deb11u10]
libnghttp2-14/oldstable-security,oldstable-security 1.43.0-1+deb11u1 amd64 [upgradable from: 1.43.0-1]
openssh-client/oldstable-security,oldstable-security 1:8.4p1-5+deb11u3 amd64 [upgradable from: 1:8.4p1-5+deb11u2]
openssh-server/oldstable-security,oldstable-security 1:8.4p1-5+deb11u3 amd64 [upgradable from: 1:8.4p1-5+deb11u2]
openssh-sftp-server/oldstable-security,oldstable-security 1:8.4p1-5+deb11u3 amd64 [upgradable from: 1:8.4p1-5+deb11u2]
postfix/oldstable-updates,oldstable-updates 3.5.23-0+deb11u1 amd64 [upgradable from: 3.5.18-0+deb11u1]
proxmox-backup-client/stable 2.4.4-1 amd64 [upgradable from: 2.4.3-1]
proxmox-backup-file-restore/stable 2.4.4-1 amd64 [upgradable from: 2.4.3-1]
pve-firmware/stable 3.6-6 all [upgradable from: 3.6-5]
tzdata/oldstable-updates,oldstable-updates 2021a-1+deb11u11 all [upgradable from: 2021a-1+deb11u10]

The direct attempt to perform an upgrade using apt list --upgradable fails with the following error message:

The following packages have unmet dependencies:
pve-kernel-5.15 : Depends: pve-kernel-5.15.131-2-pve but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

According to https://www.tecmint.com/sub-process-usr-bin-dpkg-returned-an-error-in-ubuntu/ I've tried the following commands to fix it (with root access):

1. apt --fix-broken install

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
libboost-context1.74.0 libboost-coroutine1.74.0 libboost-program-options1.74.0
libboost-thread1.74.0 libcaca0 libjemalloc2 libsdl1.2debian linux-image-5.10.0-16-amd64
linux-image-5.10.0-23-amd64 linux-image-5.10.0-24-amd64 linux-image-5.10.0-8-amd64
linux-image-5.10.0-9-amd64 pve-kernel-5.11.22-4-pve pve-kernel-5.11.22-5-pve
pve-kernel-5.15.107-2-pve pve-kernel-5.15.39-1-pve
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
pve-kernel-5.15.131-2-pve
The following NEW packages will be installed:
pve-kernel-5.15.131-2-pve
0 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
5 not fully installed or removed.
Need to get 0 B/76.8 MB of archives.
After this operation, 400 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 128379 files and directories currently installed.)
Preparing to unpack .../pve-kernel-5.15.131-2-pve_5.15.131-3_amd64.deb ...
Unpacking pve-kernel-5.15.131-2-pve (5.15.131-3) ...
dpkg: error processing archive /var/cache/apt/archives/pve-kernel-5.15.131-2-pve_5.15.131-3_amd64.deb (--unpack):
cannot copy extracted data for './boot/System.map-5.15.131-2-pve' to '/boot/System.map-5.15.131-2-pve.dpkg-new': failed to write (No space left on device)
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.15.131-2-pve /boot/vmlinuz-5.15.131-2-pve
update-initramfs: Deleting /boot/initrd.img-5.15.131-2-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.15.131-2-pve /boot/vmlinuz-5.15.131-2-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.15.131-2-pve /boot/vmlinuz-5.15.131-2-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 5.15.131-2-pve /boot/vmlinuz-5.15.131-2-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.116-1-pve
Found initrd image: /boot/initrd.img-5.15.116-1-pve
Found linux image: /boot/vmlinuz-5.15.108-1-pve
Found initrd image: /boot/initrd.img-5.15.108-1-pve
Found linux image: /boot/vmlinuz-5.15.107-2-pve
Found initrd image: /boot/initrd.img-5.15.107-2-pve
Found linux image: /boot/vmlinuz-5.15.39-1-pve
Found initrd image: /boot/initrd.img-5.15.39-1-pve
Found linux image: /boot/vmlinuz-5.10.0-26-amd64
Found linux image: /boot/vmlinuz-5.10.0-25-amd64
Found linux image: /boot/vmlinuz-5.10.0-24-amd64
Found initrd image: /boot/initrd.img-5.10.0-24-amd64
Found linux image: /boot/vmlinuz-5.10.0-23-amd64
Found initrd image: /boot/initrd.img-5.10.0-23-amd64
Found linux image: /boot/vmlinuz-5.10.0-16-amd64
Found initrd image: /boot/initrd.img-5.10.0-16-amd64
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
Errors were encountered while processing:
/var/cache/apt/archives/pve-kernel-5.15.131-2-pve_5.15.131-3_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

2. dpkg --configure -a

Setting up initramfs-tools (0.140) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-image-5.10.0-26-amd64 (5.10.197-1) ...
I: /initrd.img is now a symlink to boot/initrd.img-5.10.0-26-amd64
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.10.0-26-amd64
cat: write error: No space left on device
update-initramfs: failed for /boot/initrd.img-5.10.0-26-amd64 with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.10.0-26-amd64 (--configure):
installed linux-image-5.10.0-26-amd64 package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of linux-image-amd64:
linux-image-amd64 depends on linux-image-5.10.0-26-amd64 (= 5.10.197-1); however:
Package linux-image-5.10.0-26-amd64 is not configured yet.

dpkg: error processing package linux-image-amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
pve-kernel-5.15 depends on pve-kernel-5.15.131-2-pve; however:
Package pve-kernel-5.15.131-2-pve is not installed.

dpkg: error processing package pve-kernel-5.15 (--configure):
dependency problems - leaving unconfigured
Setting up linux-image-5.10.0-25-amd64 (5.10.191-1) ...
I: /initrd.img.old is now a symlink to boot/initrd.img-5.10.0-25-amd64
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.10.0-25-amd64
cat: write error: No space left on device
update-initramfs: failed for /boot/initrd.img-5.10.0-25-amd64 with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.10.0-25-amd64 (--configure):
installed linux-image-5.10.0-25-amd64 package post-installation script subprocess returned error exit status 1
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.15.116-1-pve
cat: write error: No space left on device
update-initramfs: failed for /boot/initrd.img-5.15.116-1-pve with 1.
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
linux-image-5.10.0-26-amd64
linux-image-amd64
pve-kernel-5.15
linux-image-5.10.0-25-amd64
initramfs-tools

Since the package in question is a kernel package, I have refrained from uninstalling it and other suggested ideas.

Attached is a complete list of versions that I have determined with the command pveversion -v.

Who has a concrete solution suggestion that will help me to successfully carry out the upgrade?

 

[Neobin]

Two problems here:

1.:
linux-image-amd64 (the default Debian kernel) conflicts with (newer versions of): pve-kernel-5.15.
- So, remove: linux-image-amd64 like mentioned here:
https://pve.proxmox.com/wiki/Upgrade_from_7_to_8#Upgrade_wants_to_remove_package_.27proxmox-ve.27

2.:

cannot copy extracted data for './boot/System.map-5.15.131-2-pve' to '/boot/System.map-5.15.131-2-pve.dpkg-new': failed to write (No space left on device)

- After removing the: linux-image-amd64 metapackage from problem 1, also remove all the Debian kernels: linux-image-*-amd64 themselves to get free space again.
- If this succeeded, run: apt full-upgrade (if this does not proceed, try: apt --fix-broken install again).
- If this succeeds without any further errors, and a(nother): apt full-upgrade shows, that you are up-to-date, reboot the PVE-host to boot with the new kernel version.
- If all is good, I would suggest to run: apt autoremove to remove some older PVE kernels to free up even more space on the boot partition.

 

[toolix]

With df -h I found out that the boot partition is 100% occupied.

Unfortunately, apt remove linux-image-amd64 did not work.

dpkg -l | egrep '^ii (pve-)?kernel-' | awk '{print $2}' | grep -v `uname -r` returned the following old kernel packages (I have hidden and kept the latest two versions)

pve-kernel-5.11
pve-kernel-5.11.22-4-pve
pve-kernel-5.11.22-5-pve
pve-kernel-5.11.22-7-pve
pve-kernel-5.15.107-2-pve

I then tried a normal removal of these packages with apt remove pve-kernel-5.11... failed to no avail.
As a result, I have successfully removed the packages as follows:

sudo dpkg --remove --force-remove-reinstreq pve-kernel-5.11
sudo dpkg --remove --force-remove-reinstreq pve-kernel-5.11.22-4-pve
sudo dpkg --remove --force-remove-reinstreq pve-kernel-5.11.22-5-pve
sudo dpkg --remove --force-remove-reinstreq pve-kernel-5.11.22-7-pve
sudo dpkg --remove --force-remove-reinstreq pve-kernel-5.15.107-2-pve

After a final apt-get autoremove I tried again apt --fix-broken install.

The last command didnt't finish completely and I decided a fresh reboot.
Unfortunately, the system (remote) has not started at all since then and I am currently working on a solution.