[SOLVED] Unable to upgrade to new kernel: no space left on device - but df says otherwise

M

meepmeep

Renowned Member
Apr 14, 2013
51
8
73
Hi

Usual upgrade today, to "Setting up pve-kernel-5.15.30-2-pve".
Unsual answer from proxmox

Code: 
Setting up pve-kernel-5.15.30-2-pve (5.15.30-3) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.30-2-pve /boot/vmlinuz-5.15.30-2-pve
run-parts: executing /etc/kernel/postinst.d/apt-dater-host_reboot 5.15.30-2-pve /boot/vmlinuz-5.15.30-2-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.30-2-pve /boot/vmlinuz-5.15.30-2-pve
update-initramfs: Generating /boot/initrd.img-5.15.30-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/FEE5-D6E7
        Copying kernel 5.11.22-1-pve
        Copying kernel 5.13.19-6-pve
        Copying kernel 5.15.30-2-pve
cp: error writing '/var/tmp/espmounts/FEE5-D6E7/initrd.img-5.15.30-2-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.30-2-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.30-2-pve (--configure):
 installed pve-kernel-5.15.30-2-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
 pve-kernel-5.15 depends on pve-kernel-5.15.30-2-pve; however:
  Package pve-kernel-5.15.30-2-pve is not configured yet.

dpkg: error processing package pve-kernel-5.15 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-5.15; however:
  Package pve-kernel-5.15 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 pve-kernel-5.15.30-2-pve
 pve-kernel-5.15
 proxmox-ve
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

No space left on device ?
df says that I have plenty of space :
Code: 
df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           32G     0   32G   0% /dev
tmpfs                         6.3G  1.5M  6.3G   1% /run
rpool/ROOT/pve-1              410G  221G  189G  54% /
tmpfs                          32G   25M   32G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
rpool                         189G  128K  189G   1% /rpool
rpool/data                    189G  256K  189G   1% /rpool/data
rpool/ROOT                    189G  128K  189G   1% /rpool/ROOT
rpool/data/subvol-156-disk-0  8.0G  1.8G  6.3G  23% /rpool/data/subvol-156-disk-0
rpool/data/subvol-102-disk-0   10G  3.9G  6.2G  39% /rpool/data/subvol-102-disk-0
rpool/data/subvol-111-disk-0  8.0G  1.4G  6.7G  17% /rpool/data/subvol-111-disk-0
rpool/data/subvol-100-disk-0   33G   12G   22G  35% /rpool/data/subvol-100-disk-0
rpool/data/subvol-157-disk-0   20G  1.3G   19G   7% /rpool/data/subvol-157-disk-0
tmpfs                         6.3G     0  6.3G   0% /run/user/1000
/dev/fuse                     128M   40K  128M   1% /etc/pve

(and all inode use level are at 1%)

What am I missing ?
Why is there missing space for '/var/tmp/espmounts/FEE5-D6E7/initrd.img-5.15.30-2-pve' but "/" has 189G available ?
 
Last edited:
Hi,

The EFI system partitions are separate partitions that need to be vfat formatted (as per EFI specification), it's normally not mounted so df doesn't list it.

You probably have a few older kernels installed, resulting in to many kernel images on the ESP, it seems you even still got 5.11 kernel images

So, for starters try
Code: 
apt purge pve-kernel-5.11
apt autoremove --purge

For a full list of installed pve kernels see the pveversion -v output.
 
  • Like
Reactions: Stoiko Ivanov
Didn't work, and I think I broke something :

Code: 
apt remove --purge pve-kernel-5.11.22-1-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  pve-kernel-5.11.22-1-pve
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
4 not fully installed or removed.
After this operation, 392 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 108867 files and directories currently installed.)
Removing pve-kernel-5.11.22-1-pve (5.11.22-2) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
update-initramfs: Deleting /boot/initrd.img-5.11.22-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/FEE5-D6E7
        Copying kernel 5.13.19-6-pve
        Copying kernel 5.15.30-2-pve
cp: error writing '/var/tmp/espmounts/FEE5-D6E7/initrd.img-5.15.30-2-pve': No space left on device
run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.11.22-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.11.22-1-pve (--remove):
 installed pve-kernel-5.11.22-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 pve-kernel-5.11.22-1-pve
Processing was halted because there were too many errors.
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Code: 
apt  --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages will be REMOVED:
  pve-kernel-5.11.22-1-pve pve-kernel-5.4 pve-kernel-5.4.124-1-pve pve-kernel-5.4.34-1-pve
0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
6 not fully installed or removed.
After this operation, 968 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 95673 files and directories currently installed.)
Removing pve-kernel-5.11.22-1-pve (5.11.22-2) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
update-initramfs: Deleting /boot/initrd.img-5.11.22-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.11.22-1-pve /boot/vmlinuz-5.11.22-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/FEE5-D6E7
        Copying kernel 5.13.19-6-pve
        Copying kernel 5.15.30-2-pve
cp: error writing '/var/tmp/espmounts/FEE5-D6E7/initrd.img-5.15.30-2-pve': No space left on device
run-parts: /etc/kernel/postrm.d/zz-proxmox-boot exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/pve-kernel-5.11.22-1-pve.postrm line 14.
dpkg: error processing package pve-kernel-5.11.22-1-pve (--remove):
 installed pve-kernel-5.11.22-1-pve package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 pve-kernel-5.11.22-1-pve
Processing was halted because there were too many errors.
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

I even have 5.4 kernel :

Code: 
dpkg -l |grep kernel |grep ii
ii  ipset                                7.10-1                         amd64        administration tool for kernel IP sets
ii  kmod                                 28-1                           amd64        tools for managing Linux kernel modules
ii  libaio1:amd64                        0.3.112-9                      amd64        Linux kernel AIO access library - shared library
ii  libdrm-common                        2.4.104-1                      all          Userspace interface to kernel DRM services -- common files
ii  libdrm2:amd64                        2.4.104-1                      amd64        Userspace interface to kernel DRM services -- runtime
ii  libnozzle1:amd64                     1.22-pve2                      amd64        userland wrapper around kernel tap devices
ii  libsctp1:amd64                       1.0.18+dfsg-1                  amd64        user-space access to Linux kernel SCTP - shared library
ii  liburing1:amd64                      0.7-3                          amd64        Linux kernel io_uring access library - shared library
ii  pve-firmware                         3.4-1                          all          Binary firmware code for the pve-kernel
ii  pve-kernel-5.11.22-7-pve             5.11.22-12                     amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.13                      7.1-9                          all          Latest Proxmox VE Kernel Image
ii  pve-kernel-5.13.19-2-pve             5.13.19-4                      amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.13.19-5-pve             5.13.19-13                     amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.13.19-6-pve             5.13.19-15                     amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.4                       6.4-4                          all          Latest Proxmox VE Kernel Image
ii  pve-kernel-5.4.124-1-pve             5.4.124-1                      amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-5.4.34-1-pve              5.4.34-2                       amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-helper                    7.2-2                          all          Function for various kernel maintenance tasks.
ii  rsyslog                              8.2102.0-2                     amd64        reliable system and kernel logging daemon

pveversion (after breaking everything)
Code: 
pveversion -v
proxmox-ve: not correctly installed (running kernel: 5.11.22-1-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.13: 7.1-9
pve-kernel-5.4: 6.4-4
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-5-pve: 5.13.19-13
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

Edit to add proxmox-boot-tool status

Code: 
 proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
FEE5-D6E7 is configured with: uefi (versions: 5.4.103-1-pve, 5.4.106-1-pve, 5.4.60-1-pve), grub (versions: 5.11.22-1-pve, 5.11.22-7-pve, 5.13.19-5-pve, 5.13.19-6-pve, 5.15.30-2-pve)
FEE6-D626 is configured with: uefi (versions: 5.4.103-1-pve, 5.4.106-1-pve, 5.4.60-1-pve), grub (versions: 5.11.22-1-pve, 5.11.22-7-pve, 5.13.19-5-pve, 5.13.19-6-pve)
FEE7-8182 is configured with: uefi (versions: 5.4.103-1-pve, 5.4.106-1-pve, 5.4.60-1-pve), grub (versions: 5.11.22-1-pve, 5.11.22-7-pve, 5.13.19-5-pve, 5.13.19-6-pve)
 
Last edited:
I think I should follow this thread : https://forum.proxmox.com/threads/dpkg-hanging-when-upgrading-pve-kernel.95077/#post-412898

But, I have 3 nvme disk on a zfs raidz configuration:

Code: 
nvme2n1     259:0    0 476.9G  0 disk
├─nvme2n1p1 259:1    0  1007K  0 part
├─nvme2n1p2 259:2    0   512M  0 part
└─nvme2n1p3 259:3    0 476.4G  0 part
nvme0n1     259:4    0 476.9G  0 disk
├─nvme0n1p1 259:5    0  1007K  0 part
├─nvme0n1p2 259:6    0   512M  0 part
└─nvme0n1p3 259:7    0 476.4G  0 part
nvme1n1     259:8    0 476.9G  0 disk
├─nvme1n1p1 259:9    0  1007K  0 part
├─nvme1n1p2 259:10   0   512M  0 part
└─nvme1n1p3 259:11   0 476.4G  0 part


I should mount nvme2n1p2, nvme0n1p2 and nvme0n1p2 individually to make some space ? Or , there is a way to mount it as a unique raidz volume ?
 
Hmm - out of curiosity - did you ever change the boot-mode of this system (from UEFI to legacy)?

could you please post the output of `lsblk -f` ?

To get out of this you could manually remove some/most of the copied kernels:
- I assume that the system is booted and always will remain booting with legacy bios and not UEFI:
- mount each of the ESPs manually
- in the mountpoint remove e.g. for kernel 5.4.103-1-pve:
- remove /mountpoint/EFI/proxmox/5.4.103-1-pve
- remove /mountpoint/loader/entries/proxmox-5.4.103-1-pve.conf

this should free up enough space already (if you remove 2 kernels+initrds)


Alternatively :
A simple and quite safe way out would be to format and reinitialize the ESPs (make sure to have a working backup of everything as usual):
* remove one of the UUIDs from /etc/kernel/proxmox-boot-uuids - should anything not work out - you should be able to boot from this disk
* for the remaining 2 - run `proxmox-boot-tool format /dev/disk/by-id/FEE6-D626 --force` and `proxmox-boot-tool init /dev/disk/by-id/FEE6-D626`
this should free up enough space to get through the upgrade


I hope this helps!
 
  • Like
Reactions: t.lamprecht
I'm a little lost on this boot-mode .. sorry.
I don't remember changing the boot-mode, but i do remember that on migration between pve 6 to 7, I did some checks with proxmox-boot-tool.

Code: 
efibootmgr -v
EFI variables are not supported on this system.
So, i'm using legacy mode, right ? but with proxmox-boot-tool managing the boot ?

Code: 
NAME        FSTYPE      FSVER    LABEL              UUID                                   FSAVAIL FSUSE% MOUNTPOINT
zd0
├─zd0p1     ext2        1.0                         f9dc7cd1-b3df-487b-ba8d-e1a1b9e616ce
├─zd0p2
└─zd0p5     LVM2_member LVM2 001                    OwkcQ1-qyD7-HxRm-79iG-mowU-0ZkT-e7mzR5
zd16
├─zd16p1    ext2        1.0                         84a21047-4cd2-4a4c-835b-4005c9dcaa66
├─zd16p2
└─zd16p5    LVM2_member LVM2 001                    Ik7dmJ-PkxT-Z5Cv-R4Oe-2UL2-Kx8v-dUBp9y
zd32
├─zd32p1    vfat        FAT12    EFISYS             BD41-11EE
├─zd32p2
└─zd32p3    ufs         2        OPNsense           5dc2fd688be24fb7
zd48
├─zd48p1    ext2        1.0                         efacd269-c794-4dbf-b3d4-243b887c5bdb
├─zd48p2
└─zd48p5    LVM2_member LVM2 001                    CXSqwT-R0s6-xNIC-XGgv-FUAW-vzoA-FHudRo
zd64
├─zd64p1    ext2        1.0                         98fd0e73-bc77-4e13-a16a-747f08c920f8
├─zd64p2
└─zd64p5    LVM2_member LVM2 001                    X7F3kt-AJdR-awad-9X7n-p8rC-k7LW-c4Pqno
zd80
├─zd80p1    ext2        1.0                         cc96c2f6-525e-4df8-b684-21bd8027bf9c
├─zd80p2
└─zd80p5    LVM2_member LVM2 001                    jFQHxF-bVTY-Cr8b-VkUY-TcpP-Z1Qz-BzmfnT
zd96
├─zd96p1    ext2        1.0                         76483cde-acf5-41d1-993c-516051946db9
├─zd96p2
└─zd96p5    LVM2_member LVM2 001                    xADAM0-8H2e-T7Qc-8KS2-ffLw-bB3l-uD6ymi
zd112
├─zd112p1   ext2        1.0                         d83147a4-4049-4682-974a-bd2364aa3228
├─zd112p2
└─zd112p5   LVM2_member LVM2 001                    MlC9BY-mlJK-54dG-14ps-gjcI-04VE-FRFiwB
zd128
├─zd128p1   ntfs                 Réservé au système E06C06A96C067A94
└─zd128p2   ntfs                                    0E74073074071A5B
zd144
├─zd144p1   ext2        1.0                         4640bf7b-bb6f-4a3d-861e-ac60f0d49fab
├─zd144p2
└─zd144p5   LVM2_member LVM2 001                    kCBfDf-hGJX-e00r-vFdz-fcfz-asDb-IfNCtw
zd160
├─zd160p1   ext2        1.0                         13fcb62e-0f58-46e8-871a-758dbf5dfa9b
├─zd160p2
└─zd160p5   LVM2_member LVM2 001                    rV0Xlv-JWxK-yQII-A9cn-RLMw-bExG-qNbe5G
zd176
├─zd176p1   vfat        FAT32                       CADF-28C4
├─zd176p2
├─zd176p3   ntfs                                    605AE0C65AE09A54
└─zd176p4   ntfs                                    723CED873CED46A7
zd192
zd208
nvme2n1
├─nvme2n1p1
├─nvme2n1p2 vfat        FAT32                       FEE7-8182
└─nvme2n1p3 zfs_member  5000     rpool              2423581133492052449
nvme0n1
├─nvme0n1p1
├─nvme0n1p2 vfat        FAT32                       FEE5-D6E7
└─nvme0n1p3 zfs_member  5000     rpool              2423581133492052449
nvme1n1
├─nvme1n1p1
├─nvme1n1p2 vfat        FAT32                       FEE6-D626
└─nvme1n1p3 zfs_member  5000     rpool              2423581133492052449
 
meepmeep said:
So, i'm using legacy mode, right ? but with proxmox-boot-tool managing the boot ?
Click to expand...
yes - would think so too :)
and yes proxmox-boot-tool manages the boots on all '/' on ZFS systems since PVE 6.4 (since grub cannot boot directly from newer zpools proxmox-boot-tool installs and configures grub on the ESPs (which are vfat))

how large are the ESPs on your system (sorry - `lsblk` has this information `lsblk -f` does not...)?

in any case I think the suggestions from above should work
 
I used this one :
Code: 
- I assume that the system is booted and always will remain booting with legacy bios and not UEFI:
- mount each of the ESPs manually
- in the mountpoint remove e.g. for kernel 5.4.103-1-pve:
- remove /mountpoint/EFI/proxmox/5.4.103-1-pve
- remove /mountpoint/loader/entries/proxmox-5.4.103-1-pve.conf

512Mo for each ESPs, and thay were almost full.

I removed all kernel on /mountpoint/EFI/proxmox/ (has it was only old 5.4.x kernel, not currently used.
apt --fix-broken install went through.
Making some backup before rebooting ;)
 
  • Like
Reactions: Stoiko Ivanov
Code: 
#proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
FEE5-D6E7 is configured with: uefi (versions: ), grub (versions: 5.13.19-6-pve, 5.15.30-2-pve)
FEE6-D626 is configured with: uefi (versions: ), grub (versions: 5.13.19-6-pve, 5.15.30-2-pve)
FEE7-8182 is configured with: uefi (versions: ), grub (versions: 5.13.19-6-pve, 5.15.30-2-pve)

# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
5.13.19-6-pve
5.15.30-2-pve
 
  • Like
Reactions: Stoiko Ivanov
Stoiko Ivanov said:
Alternatively :
A simple and quite safe way out would be to format and reinitialize the ESPs (make sure to have a working backup of everything as usual):
* remove one of the UUIDs from /etc/kernel/proxmox-boot-uuids - should anything not work out - you should be able to boot from this disk
* for the remaining 2 - run `proxmox-boot-tool format /dev/disk/by-id/FEE6-D626 --force` and `proxmox-boot-tool init /dev/disk/by-id/FEE6-D626`
this should free up enough space to get through the upgrade


I hope this helps!
Click to expand...

This worked for me, thanks!

As a note: Once I did proxmox-boot-tool format ... and proxmox-boot-tool init ..., the partitions were assigned new ID numbers, so I had to update /etc/kernel/proxmox-boot-uuids to remove the old IDs.
 
You must log in or register to reply here.
Top Bottom