[SOLVED] Kernel upgrade gets No space left on device

M

markc

Active Member
Sep 12, 2020
61
15
28
70
Gold Coast, Australia
spiderweb.com.au
Proxmox 8.0.4. I have a 4 node cluster and all of them upgraded successfully except this one with the error below. /var/tmp/espmounts/67D6-E50C/ does exist but not EFI. Any suggestion how to find out why there is no space left and how to fix it?

Code: 
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
update-initramfs: Deleting /boot/initrd.img-5.15.108-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/67D6-E50C
Copying kernel and creating boot-entry for 6.2.16-6-pve
Copying kernel and creating boot-entry for 6.2.16-8-pve

cp: error writing '/var/tmp/espmounts/67D6-E50C/EFI/proxmox/6.2.16-8-pve/initrd.img-6.2.16-8-pve': No space left on device

run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.15.108-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.15.108-1-pve (--remove):
installed pve-kernel-5.15.108-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
pve-kernel-5.15.108-1-pve
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
/var/tmp has minimal content...

Code: 
~ du -sh /var/tmp
183K    /var/tmp

I can even copy a 655 MB file into that area, larger than any kernel package...

Code: 
~ ll -h /var/tmp/espmounts/67D6-E50C
total 653M
-rw-r--r-- 1 root root 655M Sep  1 22:57 jammy-server-cloudimg-amd64.img

> of course u can increase the root partition/volume

That I do not know how to do. Any hints?

Code: 
~ df
Filesystem                                                Type      Size  Used Avail Use% Mounted on
udev                                                      devtmpfs   32G     0   32G   0% /dev
tmpfs                                                     tmpfs     6.3G  1.8M  6.3G   1% /run
rpool/ROOT/pve-1                                          zfs       1.7T  150G  1.5T   9% /
tmpfs                                                     tmpfs      32G   66M   32G   1% /dev/shm
tmpfs                                                     tmpfs     5.0M   16K  5.0M   1% /run/lock
rpool                                                     zfs       1.5T  128K  1.5T   1% /rpool
rpool/ROOT                                                zfs       1.5T  128K  1.5T   1% /rpool/ROOT
rpool/data                                                zfs       1.5T  128K  1.5T   1% /rpool/data
/dev/fuse                                                 fuse      128M   88K  128M   1% /etc/pve
tmpfs                                                     tmpfs     6.3G   56K  6.3G   1% /run/user/1001
tmpfs                                                     tmpfs      32G   28K   32G   1% /var/lib/ceph/osd/ceph-2
tmpfs                                                     tmpfs      32G   28K   32G   1% /var/lib/ceph/osd/ceph-5
192.168.20.21,192.168.20.22,192.168.20.23,192.168.20.24:/ ceph      4.5T  142G  4.4T   4% /mnt/pve/ceph-fs
tmpfs                                                     tmpfs     6.3G   32K  6.3G   1% /run/user/0
 
Thank you Neobin, this was the procedure I used...

Code: 
cat /etc/kernel/proxmox-boot-uuids
67D6-E50C
ls -al /dev/disk/by-uuid/67D6-E50C
lrwxrwxrwx 1 root root 15 Sep  1 20:06 /dev/disk/by-uuid/67D6-E50C -> ../../nvme0n1p2
mkdir /tmp/myesp
mount /dev/nvme0n1p2 /tmp/myesp
ll /tmp/myesp/*/*
rm -rf /tmp/myesp/*/*/5.15.108-1-pve
rm -rf /tmp/myesp/*/*/6.2.16-5-pve
umount /tmp/myesp
apt-get -f install
 
  • Like
Reactions: wahmed and etosolini
Thanks for the above markc.

In my case, I had multiple disks (zfs raid 10). This makes it a little easier since you need to remove old kernels from all of the disks:

Code: 
mkdir /tmp/fixme;
for i in $(cat /etc/kernel/proxmox-boot-uuids); do
    echo $i;
    mkdir /tmp/fixme/$i
    mount /dev/disk/by-uuid/$i /tmp/fixme/$i
done

Then df should show one of the UEFI partitions is full.

Poke around a bit a look at the directories that were mounted here. You should find some old kernels.
Code: 
cd /tmp/fixme

Then you can remove them with something like:
Code: 
find /tmp/fixme/ -type f -iname "*5.4*" -exec rm {} \;
 
You must log in or register to reply here.
Top Bottom