[SOLVED] Unable to upgrade to new kernel: no space left on device - but df says otherwise

meepmeep

Renowned Member
Apr 14, 2013
51
8
73
Hi

Usual upgrade today, to "Setting up pve-kernel-5.15.30-2-pve".
Unsual answer from proxmox

Code:
Setting up pve-kernel-5.15.30-2-pve (5.15.30-3) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.30-2-pve /boot/vmlinuz-5.15.30-2-pve
run-parts: executing /etc/kernel/postinst.d/apt-dater-host_reboot 5.15.30-2-pve /boot/vmlinuz-5.15.30-2-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.30-2-pve /boot/vmlinuz-5.15.30-2-pve
update-initramfs: Generating /boot/initrd.img-5.15.30-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/FEE5-D6E7
        Copying kernel 5.11.22-1-pve
        Copying kernel 5.13.19-6-pve
        Copying kernel 5.15.30-2-pve
cp: error writing '/var/tmp/espmounts/FEE5-D6E7/initrd.img-5.15.30-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.30-2-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.30-2-pve (--configure):
 installed pve-kernel-5.15.30-2-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
 pve-kernel-5.15 depends on pve-kernel-5.15.30-2-pve; however:
  Package pve-kernel-5.15.30-2-pve is not configured yet.

dpkg: error processing package pve-kernel-5.15 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-5.15; however:
  Package pve-kernel-5.15 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 pve-kernel-5.15.30-2-pve
 pve-kernel-5.15
 proxmox-ve
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

No space left on device ?
df says that I have plenty of space :
Code:
df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           32G     0   32G   0% /dev
tmpfs                         6.3G  1.5M  6.3G   1% /run
rpool/ROOT/pve-1              410G  221G  189G  54% /
tmpfs                          32G   25M   32G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
rpool                         189G  128K  189G   1% /rpool
rpool/data                    189G  256K  189G   1% /rpool/data
rpool/ROOT                    189G  128K  189G   1% /rpool/ROOT
rpool/data/subvol-156-disk-0  8.0G  1.8G  6.3G  23% /rpool/data/subvol-156-disk-0
rpool/data/subvol-102-disk-0   10G  3.9G  6.2G  39% /rpool/data/subvol-102-disk-0
rpool/data/subvol-111-disk-0  8.0G  1.4G  6.7G  17% /rpool/data/subvol-111-disk-0
rpool/data/subvol-100-disk-0   33G   12G   22G  35% /rpool/data/subvol-100-disk-0
rpool/data/subvol-157-disk-0   20G  1.3G   19G   7% /rpool/data/subvol-157-disk-0
tmpfs                         6.3G     0  6.3G   0% /run/user/1000
/dev/fuse                     128M   40K  128M   1% /etc/pve

(and all inode use level are at 1%)

What am I missing ?
Why is there missing space for '/var/tmp/espmounts/FEE5-D6E7/initrd.img-5.15.30-2-pve' but "/" has 189G available ?
 
Last edited:
Hi,

The EFI system partitions are separate partitions that need to be vfat formatted (as per EFI specification), it's normally not mounted so df doesn't list it.

You probably have a few older kernels installed, resulting in to many kernel images on the ESP, it seems you even still got 5.11 kernel images

So, for starters try
Code:
apt purge pve-kernel-5.11
apt autoremove --purge

For a full list of installed pve kernels see the pveversion -v output.
 
  • Like
Reactions: Stoiko Ivanov
Didn't work, and I think I broke something :

Code:
apt remove --purge pve-kernel-5.11.22-1-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  pve-kernel-5.11.22-1-pve
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
4 not fully installed or removed.
After this operation, 392 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 108867 files and directories currently installed.)
Removing pve-kernel-5.11.22-1-pve (5.11.22-2) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
update-initramfs: Deleting /boot/initrd.img-5.11.22-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/FEE5-D6E7
        Copying kernel 5.13.19-6-pve
        Copying kernel 5.15.30-2-pve
cp: error writing '/var/tmp/espmounts/FEE5-D6E7/initrd.img-5.15.30-2-pve': No space left on device
run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.11.22-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.11.22-1-pve (--remove):
 installed pve-kernel-5.11.22-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 pve-kernel-5.11.22-1-pve
Processing was halted because there were too many errors.
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Code:
apt  --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages will be REMOVED:
  pve-kernel-5.11.22-1-pve pve-kernel-5.4 pve-kernel-5.4.124-1-pve pve-kernel-5.4.34-1-pve
0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
6 not fully installed or removed.
After this operation, 968 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 95673 files and directories currently installed.)
Removing pve-kernel-5.11.22-1-pve (5.11.22-2) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
update-initramfs: Deleting /boot/initrd.img-5.11.22-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/FEE5-D6E7
        Copying kernel 5.13.19-6-pve
        Copying kernel 5.15.30-2-pve
cp: error writing '/var/tmp/espmounts/FEE5-D6E7/initrd.img-5.15.30-2-pve': No space left on device
run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.11.22-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.11.22-1-pve (--remove):
 installed pve-kernel-5.11.22-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 pve-kernel-5.11.22-1-pve
Processing was halted because there were too many errors.
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

I even have 5.4 kernel :

Code:
dpkg -l |grep kernel |grep ii
ii  ipset                                7.10-1                         amd64        administration tool for kernel IP sets
ii  kmod                                 28-1                           amd64        tools for managing Linux kernel modules
ii  libaio1:amd64                        0.3.112-9                      amd64        Linux kernel AIO access library - shared library
ii  libdrm-common                        2.4.104-1                      all          Userspace interface to kernel DRM services -- common files
ii  libdrm2:amd64                        2.4.104-1                      amd64        Userspace interface to kernel DRM services -- runtime
ii  libnozzle1:amd64                     1.22-pve2                      amd64        userland wrapper around kernel tap devices
ii  libsctp1:amd64                       1.0.18+dfsg-1                  amd64        user-space access to Linux kernel SCTP - shared library
ii  liburing1:amd64                      0.7-3                          amd64        Linux kernel io_uring access library - shared library
ii  pve-firmware                         3.4-1                          all          Binary firmware code for the pve-kernel
ii  pve-kernel-5.11.22-7-pve             5.11.22-12                     amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.13                      7.1-9                          all          Latest Proxmox VE Kernel Image
ii  pve-kernel-5.13.19-2-pve             5.13.19-4                      amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.13.19-5-pve             5.13.19-13                     amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.13.19-6-pve             5.13.19-15                     amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.4                       6.4-4                          all          Latest Proxmox VE Kernel Image
ii  pve-kernel-5.4.124-1-pve             5.4.124-1                      amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.4.34-1-pve              5.4.34-2                       amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-helper                    7.2-2                          all          Function for various kernel maintenance tasks.
ii  rsyslog                              8.2102.0-2                     amd64        reliable system and kernel logging daemon

pveversion (after breaking everything)
Code:
pveversion -v
proxmox-ve: not correctly installed (running kernel: 5.11.22-1-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.13: 7.1-9
pve-kernel-5.4: 6.4-4
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-5-pve: 5.13.19-13
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

Edit to add proxmox-boot-tool status

Code:
 proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
FEE5-D6E7 is configured with: uefi (versions: 5.4.103-1-pve, 5.4.106-1-pve, 5.4.60-1-pve), grub (versions: 5.11.22-1-pve, 5.11.22-7-pve, 5.13.19-5-pve, 5.13.19-6-pve, 5.15.30-2-pve)
FEE6-D626 is configured with: uefi (versions: 5.4.103-1-pve, 5.4.106-1-pve, 5.4.60-1-pve), grub (versions: 5.11.22-1-pve, 5.11.22-7-pve, 5.13.19-5-pve, 5.13.19-6-pve)
FEE7-8182 is configured with: uefi (versions: 5.4.103-1-pve, 5.4.106-1-pve, 5.4.60-1-pve), grub (versions: 5.11.22-1-pve, 5.11.22-7-pve, 5.13.19-5-pve, 5.13.19-6-pve)
 
Last edited:
I think I should follow this thread : https://forum.proxmox.com/threads/dpkg-hanging-when-upgrading-pve-kernel.95077/#post-412898

But, I have 3 nvme disk on a zfs raidz configuration:

Code:
nvme2n1     259:0    0 476.9G  0 disk
├─nvme2n1p1 259:1    0  1007K  0 part
├─nvme2n1p2 259:2    0   512M  0 part
└─nvme2n1p3 259:3    0 476.4G  0 part
nvme0n1     259:4    0 476.9G  0 disk
├─nvme0n1p1 259:5    0  1007K  0 part
├─nvme0n1p2 259:6    0   512M  0 part
└─nvme0n1p3 259:7    0 476.4G  0 part
nvme1n1     259:8    0 476.9G  0 disk
├─nvme1n1p1 259:9    0  1007K  0 part
├─nvme1n1p2 259:10   0   512M  0 part
└─nvme1n1p3 259:11   0 476.4G  0 part


I should mount nvme2n1p2, nvme0n1p2 and nvme0n1p2 individually to make some space ? Or , there is a way to mount it as a unique raidz volume ?
 
Hmm - out of curiosity - did you ever change the boot-mode of this system (from UEFI to legacy)?

could you please post the output of `lsblk -f` ?

To get out of this you could manually remove some/most of the copied kernels:
- I assume that the system is booted and always will remain booting with legacy bios and not UEFI:
- mount each of the ESPs manually
- in the mountpoint remove e.g. for kernel 5.4.103-1-pve:
- remove /mountpoint/EFI/proxmox/5.4.103-1-pve
- remove /mountpoint/loader/entries/proxmox-5.4.103-1-pve.conf

this should free up enough space already (if you remove 2 kernels+initrds)


Alternatively :
A simple and quite safe way out would be to format and reinitialize the ESPs (make sure to have a working backup of everything as usual):
* remove one of the UUIDs from /etc/kernel/proxmox-boot-uuids - should anything not work out - you should be able to boot from this disk
* for the remaining 2 - run `proxmox-boot-tool format /dev/disk/by-id/FEE6-D626 --force` and `proxmox-boot-tool init /dev/disk/by-id/FEE6-D626`
this should free up enough space to get through the upgrade


I hope this helps!
 
  • Like
Reactions: t.lamprecht
I'm a little lost on this boot-mode .. sorry.
I don't remember changing the boot-mode, but i do remember that on migration between pve 6 to 7, I did some checks with proxmox-boot-tool.

Code:
efibootmgr -v
EFI variables are not supported on this system.
So, i'm using legacy mode, right ? but with proxmox-boot-tool managing the boot ?

Code:
NAME        FSTYPE      FSVER    LABEL              UUID                                   FSAVAIL FSUSE% MOUNTPOINT
zd0
├─zd0p1     ext2        1.0                         f9dc7cd1-b3df-487b-ba8d-e1a1b9e616ce
├─zd0p2
└─zd0p5     LVM2_member LVM2 001                    OwkcQ1-qyD7-HxRm-79iG-mowU-0ZkT-e7mzR5
zd16
├─zd16p1    ext2        1.0                         84a21047-4cd2-4a4c-835b-4005c9dcaa66
├─zd16p2
└─zd16p5    LVM2_member LVM2 001                    Ik7dmJ-PkxT-Z5Cv-R4Oe-2UL2-Kx8v-dUBp9y
zd32
├─zd32p1    vfat        FAT12    EFISYS             BD41-11EE
├─zd32p2
└─zd32p3    ufs         2        OPNsense           5dc2fd688be24fb7
zd48
├─zd48p1    ext2        1.0                         efacd269-c794-4dbf-b3d4-243b887c5bdb
├─zd48p2
└─zd48p5    LVM2_member LVM2 001                    CXSqwT-R0s6-xNIC-XGgv-FUAW-vzoA-FHudRo
zd64
├─zd64p1    ext2        1.0                         98fd0e73-bc77-4e13-a16a-747f08c920f8
├─zd64p2
└─zd64p5    LVM2_member LVM2 001                    X7F3kt-AJdR-awad-9X7n-p8rC-k7LW-c4Pqno
zd80
├─zd80p1    ext2        1.0                         cc96c2f6-525e-4df8-b684-21bd8027bf9c
├─zd80p2
└─zd80p5    LVM2_member LVM2 001                    jFQHxF-bVTY-Cr8b-VkUY-TcpP-Z1Qz-BzmfnT
zd96
├─zd96p1    ext2        1.0                         76483cde-acf5-41d1-993c-516051946db9
├─zd96p2
└─zd96p5    LVM2_member LVM2 001                    xADAM0-8H2e-T7Qc-8KS2-ffLw-bB3l-uD6ymi
zd112
├─zd112p1   ext2        1.0                         d83147a4-4049-4682-974a-bd2364aa3228
├─zd112p2
└─zd112p5   LVM2_member LVM2 001                    MlC9BY-mlJK-54dG-14ps-gjcI-04VE-FRFiwB
zd128
├─zd128p1   ntfs                 Réservé au système E06C06A96C067A94
└─zd128p2   ntfs                                    0E74073074071A5B
zd144
├─zd144p1   ext2        1.0                         4640bf7b-bb6f-4a3d-861e-ac60f0d49fab
├─zd144p2
└─zd144p5   LVM2_member LVM2 001                    kCBfDf-hGJX-e00r-vFdz-fcfz-asDb-IfNCtw
zd160
├─zd160p1   ext2        1.0                         13fcb62e-0f58-46e8-871a-758dbf5dfa9b
├─zd160p2
└─zd160p5   LVM2_member LVM2 001                    rV0Xlv-JWxK-yQII-A9cn-RLMw-bExG-qNbe5G
zd176
├─zd176p1   vfat        FAT32                       CADF-28C4
├─zd176p2
├─zd176p3   ntfs                                    605AE0C65AE09A54
└─zd176p4   ntfs                                    723CED873CED46A7
zd192
zd208
nvme2n1
├─nvme2n1p1
├─nvme2n1p2 vfat        FAT32                       FEE7-8182
└─nvme2n1p3 zfs_member  5000     rpool              2423581133492052449
nvme0n1
├─nvme0n1p1
├─nvme0n1p2 vfat        FAT32                       FEE5-D6E7
└─nvme0n1p3 zfs_member  5000     rpool              2423581133492052449
nvme1n1
├─nvme1n1p1
├─nvme1n1p2 vfat        FAT32                       FEE6-D626
└─nvme1n1p3 zfs_member  5000     rpool              2423581133492052449
 
So, i'm using legacy mode, right ? but with proxmox-boot-tool managing the boot ?
yes - would think so too :)
and yes proxmox-boot-tool manages the boots on all '/' on ZFS systems since PVE 6.4 (since grub cannot boot directly from newer zpools proxmox-boot-tool installs and configures grub on the ESPs (which are vfat))

how large are the ESPs on your system (sorry - `lsblk` has this information `lsblk -f` does not...)?

in any case I think the suggestions from above should work
 
I used this one :
Code:
- I assume that the system is booted and always will remain booting with legacy bios and not UEFI:
- mount each of the ESPs manually
- in the mountpoint remove e.g. for kernel 5.4.103-1-pve:
- remove /mountpoint/EFI/proxmox/5.4.103-1-pve
- remove /mountpoint/loader/entries/proxmox-5.4.103-1-pve.conf

512Mo for each ESPs, and thay were almost full.

I removed all kernel on /mountpoint/EFI/proxmox/ (has it was only old 5.4.x kernel, not currently used.
apt --fix-broken install went through.
Making some backup before rebooting ;)
 
  • Like
Reactions: Stoiko Ivanov
Code:
#proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
FEE5-D6E7 is configured with: uefi (versions: ), grub (versions: 5.13.19-6-pve, 5.15.30-2-pve)
FEE6-D626 is configured with: uefi (versions: ), grub (versions: 5.13.19-6-pve, 5.15.30-2-pve)
FEE7-8182 is configured with: uefi (versions: ), grub (versions: 5.13.19-6-pve, 5.15.30-2-pve)

# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
5.13.19-6-pve
5.15.30-2-pve
 
  • Like
Reactions: Stoiko Ivanov
Alternatively :
A simple and quite safe way out would be to format and reinitialize the ESPs (make sure to have a working backup of everything as usual):
* remove one of the UUIDs from /etc/kernel/proxmox-boot-uuids - should anything not work out - you should be able to boot from this disk
* for the remaining 2 - run `proxmox-boot-tool format /dev/disk/by-id/FEE6-D626 --force` and `proxmox-boot-tool init /dev/disk/by-id/FEE6-D626`
this should free up enough space to get through the upgrade


I hope this helps!

This worked for me, thanks!

As a note: Once I did proxmox-boot-tool format ... and proxmox-boot-tool init ..., the partitions were assigned new ID numbers, so I had to update /etc/kernel/proxmox-boot-uuids to remove the old IDs.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!