apt upgrade fails. No space left on device

J

juanjfdez

Member
Mar 17, 2022
4
0
6
54
Hi all, apparently the EFI Partition at /dev/sda2 y /dev/sdb2 its 512 MB are full with old kernels therefore I can not do anymore apt upgrades and neither I can apt purge those old kernel packages either. This is 2 disk, 1 TB each, system, as a ZFS Mirror pool.

Manually on /boot there are quite a few old kernels and I can remove them with simple rm but /boot/efi shows nothing . On the contrary /boot/grub has the normal stuff, my System is able to boot in both modes , BIOS and UEFI and normally I boot it in UEFI which uses systemd-boot as per PVE default install.

The command dpkg -l | grep 'kernel*.-*'list them all and the command proxmox-boot-tool list and status shows the last 3 kernels as expected so my question is How do I manage to clear and clean that /dev/sda2 and sdb2 partitions which are vFAT type by the way?.

After removing with rm manually 2 or 3 old kernels and still having the same issue that no apt purge or dpkg purge cpmmand will work I decided to go ahead and use the FORMAT and INIT subcommands of the proxmox-boot-tool but I only did it on /dev/sdb2 just in case. So I did FORMAT with --force and it worked :

proxmox-boot-tool format --force /dev/sdb2

But it obviously generated or created new UUID apparently but wen I do the INIT subcommand it complains that cant find bootctl or something like there is no systemd-boot installed. The command apt list systemd-boot does not show it installed BUT I can not install anything anyways. So in other words the next INIT command fails:

proxmox-boot-tool init /dev/sdb2

And therefore I can not do the necessary REFRESH subcommand to finish the process as per Proxmox Host Bootloader online documentation

proxmox-boot-tool refresh

I dont think I have to do update-grub in this case right? because it is the UEFI systemd-boot start up (black screen with only 2 or 3 while lines to select either kernel to boot) and therefore refresh subcommand does that for UEFI systems.

comands like df -h /boot/efi obviously show plenty of disk space on the zpool rpool and I can even copy large 1 GB files to /boot for instance.

The issue is really the /dev/sda2 and sdb2 vFAT partitions are full with 512 MB. So I need advice to clear or clean them or remove some old stuff off it and BTW I regularly always use apt clean and also apt autoremove --purge to remove old kernels so no idea why this Debian system kept that many old kernels in the first place

Thanks in advance for any advice or help
Cheers
JJ
 
Hi juanjfdez!

If the command to df -h /boot/efi shows how much space is left on the ZFS rpool, then it probably means that /dev/sda2 nor /dev/sdb2 is not mounted yet. This is as it should be to prevent any corruption introduced by accidental writes or system crashes to this partition.

Could you temporarily mount both /dev/sda2 and /dev/sdb2 and post the output of df -h and ls -lha on these? And if possible, could you provide the output of dpkg -l | grep kernel and a more detailed error messages when running the above mentioned commands (purge, init, ...)?

I guess/suspect that the system has many kernels from different series (e.g. 5.13, 5.15, 6.4, 6.8, ...) installed, which are all kept at the moment and not marked as "auto-removable" [0], which fills up the ESP partition quite a lot.

[0] https://git.proxmox.com/?p=proxmox-...2c73e0ee3a0ab5218e20;hb=refs/heads/master#l20
 
dakralex said:
Hi juanjfdez!

If the command to df -h /boot/efi shows how much space is left on the ZFS rpool, then it probably means that /dev/sda2 nor /dev/sdb2 is not mounted yet. This is as it should be to prevent any corruption introduced by accidental writes or system crashes to this partition.

Could you temporarily mount both /dev/sda2 and /dev/sdb2 and post the output of df -h and ls -lha on these? And if possible, could you provide the output of dpkg -l | grep kernel and a more detailed error messages when running the above mentioned commands (purge, init, ...)?

I guess/suspect that the system has many kernels from different series (e.g. 5.13, 5.15, 6.4, 6.8, ...) installed, which are all kept at the moment and not marked as "auto-removable" [0], which fills up the ESP partition quite a lot.

[0] https://git.proxmox.com/?p=proxmox-...2c73e0ee3a0ab5218e20;hb=refs/heads/master#l20
Click to expand...
Thanks Daniel for the prompt reply despite the time zones difference. this is the info you ask (now before I do anything):

root@nest:~# df -h /dev/sda2
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
root@nest:~# df -h /dev/sdb2
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
root@nest:~# df -h /boot/efi
Filesystem Size Used Avail Use% Mounted on
rpool/ROOT/pve-1 857G 20G 838G 3% /

Yep , Now After I mounted sda2 it confirms it is 512 MB full.

root@nest:~# df -h /dev/sda2
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 511M 511M 0 100% /boot/efi
root@nest:~# ls -lah /dev/sda2
brw-rw---- 1 root disk 8, 2 Nov 17 21:06 /dev/sda2

And Yep the next is consistent with what systemd-boot-tool was reporting: (although I thought I would see more old kernels?)

root@nest:~# ls -lah /boot/efi
total 290M
drwxr-xr-x 5 root root 4.0K Dec 31 1969 .
drwxr-xr-x 5 root root 43 Nov 19 20:22 ..
drwxr-xr-x 6 root root 4.0K Dec 17 2020 EFI
drwxr-xr-x 5 root root 4.0K Nov 23 2023 grub
-rwxr-xr-x 1 root root 58M Nov 23 2023 initrd.img-6.2.16-19-pve
-rwxr-xr-x 1 root root 58M Nov 23 2023 initrd.img-6.5.11-4-pve
-rwxr-xr-x 1 root root 58M Jul 27 13:02 initrd.img-6.5.13-6-pve
-rwxr-xr-x 1 root root 58M Aug 31 12:43 initrd.img-6.8.12-1-pve
drwxr-xr-x 3 root root 4.0K Sep 19 11:32 loader
-rwxr-xr-x 1 root root 13M Oct 24 2023 vmlinuz-6.2.16-19-pve
-rwxr-xr-x 1 root root 13M Nov 20 2023 vmlinuz-6.5.11-4-pve
-rwxr-xr-x 1 root root 13M Jul 26 08:34 vmlinuz-6.5.13-6-pve
-rwxr-xr-x 1 root root 14M Aug 5 12:17 vmlinuz-6.8.12-1-pve
-rwxr-xr-x 1 root root 0 Nov 17 20:35 vmlinuz-6.8.12-2-pve
-rwxr-xr-x 1 root root 8.2M Sep 16 15:32 vmlinuz-6.8.8-4-pve

Now as per your request I will UnMount /devsda2 and then try to mount /dev/sdb2 but probably because I had FORMAT it earlier then it wont mount or show anything yet I guess ... and Yep , as expected it shows nothing or shows almost empty as I could not complete the INIT subcommand the other day ....

root@nest:~# df -h /dev/sdb2
Filesystem Size Used Avail Use% Mounted on
/dev/sdb2 511M 12K 511M 1% /boot/efi

root@nest:~# ls -lah /boot/efi
total 17K
drwxr-xr-x 3 root root 4.0K Dec 31 1969 .
drwxr-xr-x 5 root root 43 Nov 19 20:22 ..
drwxr-xr-x 3 root root 4.0K Nov 19 20:47 EFI

The next is the CURRENT output of dpkg kernel ... just the portion with only kernel stuff ... As you can see I had already tried to purge 6.5.11-1 and 5.4.73-1 therefore they are marked for removal I guess
ii proxmox-default-kernel 1.1.0 all Default Proxmox Kernel Image
ii proxmox-kernel-6.5 6.5.13-6 all Latest Proxmox Kernel Image
rH proxmox-kernel-6.5.11-1-pve 6.5.11-1 amd64 Proxmox Kernel Image
iF proxmox-kernel-6.5.11-2-pve 6.5.11-2 amd64 Proxmox Kernel Image
iF proxmox-kernel-6.5.11-3-pve 6.5.11-3 amd64 Proxmox Kernel Image
ii proxmox-kernel-6.5.13-6-pve-signed 6.5.13-6 amd64 Proxmox Kernel Image (signed)
iU proxmox-kernel-6.8 6.8.12-4 all Latest Proxmox Kernel Image
ii proxmox-kernel-6.8.12-1-pve-signed 6.8.12-1 amd64 Proxmox Kernel Image (signed)
iF proxmox-kernel-6.8.12-2-pve-signed 6.8.12-2 amd64 Proxmox Kernel Image (signed)
iF proxmox-kernel-6.8.12-4-pve-signed 6.8.12-4 amd64 Proxmox Kernel Image (signed)
ii proxmox-kernel-6.8.8-4-pve-signed 6.8.8-4 amd64 Proxmox Kernel Image (signed)
ii proxmox-kernel-helper 8.1.0 all Function for various kernel maintenance tasks.
ii pve-firmware 3.14-1 all Binary firmware code for the pve-kernel
ii pve-kernel-5.13.19-2-pve 5.13.19-4 amd64 The Proxmox PVE Kernel Image
rc pve-kernel-5.15.53-1-pve 5.15.53-1 amd64 Proxmox Kernel Image
ii pve-kernel-5.4 6.4-4 all Latest Proxmox VE Kernel Image
ii pve-kernel-5.4.119-1-pve 5.4.119-1 amd64 The Proxmox PVE Kernel Image
ii pve-kernel-5.4.124-1-pve 5.4.124-1 amd64 The Proxmox PVE Kernel Image
rH pve-kernel-5.4.73-1-pve 5.4.73-1 amd64 The Proxmox PVE Kernel Image
rc pve-kernel-helper 7.3-6 all Function for various kernel maintenance tasks.

Now this is the error I get when I try the INIT subcommand on the NOW UNMOUNTED /dev/sdb2
root@nest:~# proxmox-boot-tool init /dev/sdb2
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
UUID="F8F3-6927" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="sdb" MOUNTPOINT=""
Mounting '/dev/sdb2' on '/var/tmp/espmounts/F8F3-6927'.
Installing systemd-boot..
E: bootctl is not available - make sure systemd-boot is installed

Now I will post the error message I get when I try to do a simple apt command that involves rebuilding kernel stuff, say apt upgrade -y and Thanks so Much for the help BTW. I will be waiting for more instructions. Cheers JJ

PS: Kernel 5.4.73-1-pve is still marked for removal because 2 days ago I tried apt purge (that package)

root@nest:~# apt upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be REMOVED:
pve-kernel-5.4.73-1-pve
The following NEW packages will be installed:
libpve-network-api-perl
The following packages will be upgraded:
ifupdown2 libjs-extjs libpve-access-control libpve-cluster-api-perl libpve-cluster-perl libpve-common-perl libpve-guest-common-perl libpve-network-perl libpve-notify-perl libpve-rs-perl libpve-storage-perl
proxmox-backup-client proxmox-backup-file-restore proxmox-firewall proxmox-mail-forward proxmox-ve proxmox-widget-toolkit pve-cluster pve-container pve-docs pve-firewall pve-ha-manager pve-i18n pve-manager
qemu-server
25 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
8 not fully installed or removed.
Need to get 21.1 MB/138 MB of archives.
After this operation, 284 MB disk space will be freed.
Get:1 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 libpve-rs-perl amd64 0.9.0 [2,799 kB]
Get:2 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 libpve-storage-perl all 8.2.9 [146 kB]
Get:3 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 pve-firewall amd64 5.1.0 [75.2 kB]
Get:4 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 qemu-server amd64 8.3.0 [289 kB]
Get:5 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 pve-ha-manager amd64 4.0.6 [65.6 kB]
Get:6 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 pve-docs all 8.3.1 [15.6 MB]
Get:7 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 pve-i18n all 3.3.1 [412 kB]
Get:8 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 pve-manager all 8.3.0 [548 kB]
Get:9 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 libpve-network-perl all 0.10.0 [36.5 kB]
Get:10 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 libpve-network-api-perl all 0.10.0 [13.7 kB]
Get:11 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 proxmox-firewall amd64 0.6.0 [1,042 kB]
Get:12 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 proxmox-ve all 8.3.0 [5,228 B]
Fetched 21.1 MB in 1s (16.2 MB/s)
apt-listchanges: Reading changelogs...
(Reading database ... 177525 files and directories currently installed.)
Removing pve-kernel-5.4.73-1-pve (5.4.73-1) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.4.73-1-pve /boot/vmlinuz-5.4.73-1-pve
update-initramfs: Deleting /boot/initrd.img-5.4.73-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.4.73-1-pve /boot/vmlinuz-5.4.73-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.4.73-1-pve /boot/vmlinuz-5.4.73-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/1808-B094
Copying kernel and creating boot-entry for 6.5.13-6-pve
Copying kernel and creating boot-entry for 6.8.12-1-pve
Copying kernel and creating boot-entry for 6.8.12-2-pve
mkdir: cannot create directory ‘/var/tmp/espmounts/1808-B094/EFI/proxmox/6.8.12-2-pve’: No space left on device
run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.4.73-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.4.73-1-pve (--remove):
installed pve-kernel-5.4.73-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
pve-kernel-5.4.73-1-pve
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back