[SOLVED] Kernel upgrade gets No space left on device

markc · Sep 1, 2023

Proxmox 8.0.4. I have a 4 node cluster and all of them upgraded successfully except this one with the error below. /var/tmp/espmounts/67D6-E50C/ does exist but not EFI. Any suggestion how to find out why there is no space left and how to fix it?

Code:

run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
update-initramfs: Deleting /boot/initrd.img-5.15.108-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/67D6-E50C
Copying kernel and creating boot-entry for 6.2.16-6-pve
Copying kernel and creating boot-entry for 6.2.16-8-pve

cp: error writing '/var/tmp/espmounts/67D6-E50C/EFI/proxmox/6.2.16-8-pve/initrd.img-6.2.16-8-pve': No space left on device

run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.15.108-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.15.108-1-pve (--remove):
installed pve-kernel-5.15.108-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
pve-kernel-5.15.108-1-pve
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)

floh8 · Sep 1, 2023

markc said:
why there is no space

because its full

markc said:
how to fix it

u have to delete something. u can use the command "du" to find "big" directories. simple steps are to autoremove old packages or clean /tmp.
of course u can increase the root partition/volume

Neobin · Sep 1, 2023

Maybe these help:

markc · Sep 1, 2023

/var/tmp has minimal content...

Code:

~ du -sh /var/tmp
183K    /var/tmp

I can even copy a 655 MB file into that area, larger than any kernel package...

Code:

~ ll -h /var/tmp/espmounts/67D6-E50C
total 653M
-rw-r--r-- 1 root root 655M Sep  1 22:57 jammy-server-cloudimg-amd64.img

> of course u can increase the root partition/volume

That I do not know how to do. Any hints?

Code:

~ df
Filesystem                                                Type      Size  Used Avail Use% Mounted on
udev                                                      devtmpfs   32G     0   32G   0% /dev
tmpfs                                                     tmpfs     6.3G  1.8M  6.3G   1% /run
rpool/ROOT/pve-1                                          zfs       1.7T  150G  1.5T   9% /
tmpfs                                                     tmpfs      32G   66M   32G   1% /dev/shm
tmpfs                                                     tmpfs     5.0M   16K  5.0M   1% /run/lock
rpool                                                     zfs       1.5T  128K  1.5T   1% /rpool
rpool/ROOT                                                zfs       1.5T  128K  1.5T   1% /rpool/ROOT
rpool/data                                                zfs       1.5T  128K  1.5T   1% /rpool/data
/dev/fuse                                                 fuse      128M   88K  128M   1% /etc/pve
tmpfs                                                     tmpfs     6.3G   56K  6.3G   1% /run/user/1001
tmpfs                                                     tmpfs      32G   28K   32G   1% /var/lib/ceph/osd/ceph-2
tmpfs                                                     tmpfs      32G   28K   32G   1% /var/lib/ceph/osd/ceph-5
192.168.20.21,192.168.20.22,192.168.20.23,192.168.20.24:/ ceph      4.5T  142G  4.4T   4% /mnt/pve/ceph-fs
tmpfs                                                     tmpfs     6.3G   32K  6.3G   1% /run/user/0

markc · Sep 1, 2023

Thank you Neobin, this was the procedure I used...

Code:

cat /etc/kernel/proxmox-boot-uuids
67D6-E50C
ls -al /dev/disk/by-uuid/67D6-E50C
lrwxrwxrwx 1 root root 15 Sep  1 20:06 /dev/disk/by-uuid/67D6-E50C -> ../../nvme0n1p2
mkdir /tmp/myesp
mount /dev/nvme0n1p2 /tmp/myesp
ll /tmp/myesp/*/*
rm -rf /tmp/myesp/*/*/5.15.108-1-pve
rm -rf /tmp/myesp/*/*/6.2.16-5-pve
umount /tmp/myesp
apt-get -f install

soapee01 · Feb 12, 2025

Thanks for the above markc.

In my case, I had multiple disks (zfs raid 10). This makes it a little easier since you need to remove old kernels from all of the disks:

Code:

mkdir /tmp/fixme;
for i in $(cat /etc/kernel/proxmox-boot-uuids); do
    echo $i;
    mkdir /tmp/fixme/$i
    mount /dev/disk/by-uuid/$i /tmp/fixme/$i
done

Then df should show one of the UEFI partitions is full.

Poke around a bit a look at the directories that were mounted here. You should find some old kernels.

Code:

cd /tmp/fixme

Then you can remove them with something like:

Code:

find /tmp/fixme/ -type f -iname "*5.4*" -exec rm {} \;

mnih · 2026-06-21T23:37:59+0200

I got the same problem today, and while I could solve it thanks to this thread, I am a bit lost as to why this problem occurs and how this relates to the files in /boot.

problem:

Code:

Setting up proxmox-kernel-7.0.12-1-pve-signed (7.0.12-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 7.0.12-1-pve /boot/vmlinuz-7.0.12-1-pve
update-initramfs: Generating /boot/initrd.img-7.0.12-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/3290-3D0E
    Copying kernel 6.17.13-13-pve
    Copying kernel 7.0.12-1-pve
cp: error writing '/var/tmp/espmounts/3290-3D0E/initrd.img-7.0.12-1-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-7.0.12-1-pve-signed.postinst line 20.
dpkg: error processing package proxmox-kernel-7.0.12-1-pve-signed (--configure):
 installed proxmox-kernel-7.0.12-1-pve-signed package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-kernel-7.0:
 proxmox-kernel-7.0 depends on proxmox-kernel-7.0.12-1-pve-signed | proxmox-kernel-7.0.12-1-pve; however:
  Package proxmox-kernel-7.0.12-1-pve-signed is not configured yet.
  Package proxmox-kernel-7.0.12-1-pve is not installed.
  Package proxmox-kernel-7.0.12-1-pve-signed which provides proxmox-kernel-7.0.12-1-pve is not configured yet.

dpkg: error processing package proxmox-kernel-7.0 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 proxmox-kernel-7.0.12-1-pve-signed
 proxmox-kernel-7.0
E: Sub-process /usr/bin/dpkg returned an error code (1)

solved via deleting some files from the esp partition:

Code:

root@torte:~# ll /tmp/myesp/*7.0.2-6*
-rwxr-xr-x 1 root root 88M May 21 22:46 /tmp/myesp/initrd.img-7.0.2-6-pve
-rwxr-xr-x 1 root root 16M May 20 10:55 /tmp/myesp/vmlinuz-7.0.2-6-pve
root@torte:~# rm /tmp/myesp/*7.0.2-6*
root@torte:~# ll /tmp/myesp/*7.0.2-6*
ls: cannot access '/tmp/myesp/*7.0.2-6*': No such file or directory
root@torte:~# ll /tmp/myesp/*7.0.2-7*
-rwxr-xr-x 1 root root 88M May 28 22:47 /tmp/myesp/initrd.img-7.0.2-7-pve
-rwxr-xr-x 1 root root 16M May 28 10:16 /tmp/myesp/vmlinuz-7.0.2-7-pve
root@torte:~# rm /tmp/myesp/*7.0.2-7*

dpkg worked now and the bootloader update process showed me some even older 5.11 kernels back from 2021.

looking at /boot I still found the same entries I just deleted from the esp partition and also some older ones:

Code:

root@torte:~# ll /boot/
total 738M
drwxr-xr-x  5 root root   41 Jun 21 23:29 .
drwxr-xr-x 21 root root   25 Feb 15  2024 ..
-rw-r--r--  1 root root 247K Sep 22  2021 config-5.11.22-4-pve
-rw-r--r--  1 root root 247K Nov  7  2021 config-5.11.22-7-pve
-rw-r--r--  1 root root 251K Nov 29  2021 config-5.13.19-2-pve
-rw-r--r--  1 root root 296K May 29 12:08 config-6.17.13-13-pve
-rw-r--r--  1 root root 302K Jun  9 23:07 config-7.0.12-1-pve
-rw-r--r--  1 root root 302K May 20 10:55 config-7.0.2-6-pve
-rw-r--r--  1 root root 302K May 28 10:16 config-7.0.2-7-pve
-rw-r--r--  1 root root 302K May 29 13:08 config-7.0.6-2-pve
drwxr-xr-x  2 root root    2 Oct 29  2021 efi
drwxr-xr-x  2 root root    5 Jun 21 23:29 grub
-rw-r--r--  1 root root  56M Oct 29  2021 initrd.img-5.11.22-4-pve
-rw-r--r--  1 root root  56M Nov 11  2021 initrd.img-5.11.22-7-pve
-rw-r--r--  1 root root  57M Dec  5  2021 initrd.img-5.13.19-2-pve
-rw-r--r--  1 root root  86M May 29 22:48 initrd.img-6.17.13-13-pve
-rw-r--r--  1 root root  88M Jun 21 23:29 initrd.img-7.0.12-1-pve
-rw-r--r--  1 root root  88M May 21 22:46 initrd.img-7.0.2-6-pve
-rw-r--r--  1 root root  88M May 28 22:47 initrd.img-7.0.2-7-pve
-rw-r--r--  1 root root  88M May 29 22:48 initrd.img-7.0.6-2-pve
-rw-r--r--  1 root root 148K Nov 17  2024 memtest86+ia32.bin
-rw-r--r--  1 root root 149K Nov 17  2024 memtest86+ia32.efi
-rw-r--r--  1 root root 153K Nov 17  2024 memtest86+x64.bin
-rw-r--r--  1 root root 154K Nov 17  2024 memtest86+x64.efi
drwxr-xr-x  2 root root   20 Jun 21 23:29 pve
-rw-r--r--  1 root root 5.5M Sep 22  2021 System.map-5.11.22-4-pve
-rw-r--r--  1 root root 5.5M Nov  7  2021 System.map-5.11.22-7-pve
-rw-r--r--  1 root root 5.6M Nov 29  2021 System.map-5.13.19-2-pve
-rw-r--r--  1 root root 9.0M May 29 12:08 System.map-6.17.13-13-pve
-rw-r--r--  1 root root  10M Jun  9 23:07 System.map-7.0.12-1-pve
-rw-r--r--  1 root root  10M May 20 10:55 System.map-7.0.2-6-pve
-rw-r--r--  1 root root  10M May 28 10:16 System.map-7.0.2-7-pve
-rw-r--r--  1 root root  10M May 29 13:08 System.map-7.0.6-2-pve
-rw-r--r--  1 root root  14M Sep 22  2021 vmlinuz-5.11.22-4-pve
-rw-r--r--  1 root root  14M Nov  7  2021 vmlinuz-5.11.22-7-pve
-rw-r--r--  1 root root 9.6M Nov 29  2021 vmlinuz-5.13.19-2-pve
-rw-r--r--  1 root root  16M May 29 12:08 vmlinuz-6.17.13-13-pve
-rw-r--r--  1 root root  16M Jun  9 23:07 vmlinuz-7.0.12-1-pve
-rw-r--r--  1 root root  16M May 20 10:55 vmlinuz-7.0.2-6-pve
-rw-r--r--  1 root root  16M May 28 10:16 vmlinuz-7.0.2-7-pve
-rw-r--r--  1 root root  16M May 29 13:08 vmlinuz-7.0.6-2-pve

does this relate somehow? Is there a way to fix this problem for good? (right now, if the esp partition could possible be full again on the next kernel release to my understanding)

[SOLVED] Kernel upgrade gets No space left on device

markc

Active Member

floh8

Renowned Member

Neobin

Distinguished Member

markc

Active Member

markc

Active Member

soapee01

Renowned Member

mnih

Well-Known Member

We value your privacy