7.3.6 Apt-dist-upgrade failure

atticusghost

New Member
Feb 24, 2023
7
0
1
Good morning. In an attempt to patch one of my VE hosts up to the latest version. It failed to update the kernel correctly. I have yet to reboot as im afraid it wont come back up and im remote right now

Here is the logs from the Dist-Upgrade. Output for apt --fix-broken-install is also below. Im not sure if something is wrong with the current version based on the mkdir failure for that kernel or not.

Code:
root@RAMPLH1:~# apt-get dist-upgrade
Current default time zone: 'America/Toronto'
Local time is now:      Sat Jan 27 09:27:09 EST 2024.
Universal Time is now:  Sat Jan 27 14:27:09 UTC 2024.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

Setting up pve-cluster (7.3-3) ...
Setting up mariadb-common (1:10.5.21-0+deb11u1) ...
Setting up grub-common (2.06-3~deb11u6) ...
Setting up libblas3:amd64 (3.9.0-3+deb11u1) ...
Setting up libunbound8:amd64 (1.13.1-1+deb11u1) ...
Setting up proxmox-backup-client (2.4.4-1) ...
Setting up libx11-data (2:1.7.2-1+deb11u2) ...
Setting up libpve-rs-perl (0.7.7) ...
Setting up libavahi-common-data:amd64 (0.8-5+deb11u2) ...
Setting up libncurses6:amd64 (6.2+20201114-2+deb11u2) ...
Setting up spl (2.1.14-pve1) ...
Setting up libdbus-1-3:amd64 (1.12.28-0+deb11u1) ...
Setting up dbus (1.12.28-0+deb11u1) ...
A reboot is required to replace the running dbus-daemon.
Please reboot the system when convenient.
dbus.service is a disabled or a static unit, not starting it.
dbus.socket is a disabled or a static unit, not starting it.
Setting up pve-kernel-5.15.131-2-pve (5.15.131-3) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.131-2-pve /boo                                                                                 t/vmlinuz-5.15.131-2-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.131-2-pve /boot                                                                                 /vmlinuz-5.15.131-2-pve
update-initramfs: Generating /boot/initrd.img-5.15.131-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount names                                                                                 pace..
Copying and configuring kernels on /dev/disk/by-uuid/5881-7511
        Copying kernel and creating boot-entry for 5.15.102-1-pve
        Copying kernel and creating boot-entry for 5.15.131-2-pve
        Removing old version 5.15.74-1-pve
        Removing old version 5.15.85-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/5882-4B99
        Copying kernel and creating boot-entry for 5.15.102-1-pve
        Copying kernel and creating boot-entry for 5.15.131-2-pve
        Removing old version 5.15.74-1-pve
rm: cannot remove '/var/tmp/espmounts/5882-4B99/EFI/proxmox/5.15.74-1-pve/initrd                                                                                 .img-5.15.74-1-pve': Read-only file system
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return co                                                                                 de 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.1                                                                                 31-2-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.131-2-pve (--configure):
 installed pve-kernel-5.15.131-2-pve package post-installation script subprocess                                                                                  returned error exit status 2
Setting up udev (247.3-7+1-pmx11u1) ...
Setting up usb.ids (2023.01.16-0+deb11u1) ...
Setting up libwebp6:amd64 (0.6.1-2.1+deb11u2) ...
Setting up libmariadb3:amd64 (1:10.5.21-0+deb11u1) ...
Setting up pve-i18n (2.12-1) ...
Setting up libncursesw6:amd64 (6.2+20201114-2+deb11u2) ...
Setting up postfix (3.5.23-0+deb11u1) ...

Postfix (main.cf) configuration was untouched.  If you need to make changes,
edit /etc/postfix/main.cf (and others) as needed.  To view Postfix
configuration values, see postconf(1).

After modifying main.cf, be sure to run 'systemctl reload postfix'.

Setting up liblua5.3-0:amd64 (5.3.3-1.1+deb11u1) ...
Setting up libpve-cluster-perl (7.3-3) ...
Setting up libcurl4:amd64 (7.74.0-1.3+deb11u11) ...
Setting up libx11-6:amd64 (2:1.7.2-1+deb11u2) ...
Setting up isc-dhcp-common (4.4.1-2.3+deb11u2) ...
Setting up libtiff5:amd64 (4.2.0-1+deb11u5) ...
Setting up curl (7.74.0-1.3+deb11u11) ...
Setting up libuutil3linux (2.1.14-pve1) ...
Setting up python3-protobuf (3.12.4-1+deb11u1) ...
Setting up libpve-http-server-perl (4.2-3) ...
Setting up grub-efi-amd64-bin (2.06-3~deb11u6) ...
Setting up openssl (1.1.1w-0+deb11u1) ...
Setting up libbsd0:amd64 (0.11.3-1+deb11u1) ...
Setting up pve-edk2-firmware (3.20230228-4~bpo11+1) ...
Setting up libjson-c5:amd64 (0.15-2+deb11u1) ...
Setting up libxml2:amd64 (2.9.10+dfsg-6.7+deb11u4) ...
Setting up libgtk-3-common (3.24.24-4+deb11u3) ...
Setting up ncurses-term (6.2+20201114-2+deb11u2) ...
Setting up openssh-sftp-server (1:8.4p1-5+deb11u3) ...
Setting up libx11-xcb1:amd64 (2:1.7.2-1+deb11u2) ...
Setting up systemd-sysv (247.3-7+1-pmx11u1) ...
Setting up openssh-server (1:8.4p1-5+deb11u3) ...
rescue-ssh.target is a disabled or a static unit, not starting it.
Setting up bind9-libs:amd64 (1:9.16.44-1~deb11u1) ...
Setting up grub2-common (2.06-3~deb11u6) ...
Setting up pve-qemu-kvm (7.2.0-8) ...
Setting up libavahi-common3:amd64 (0.8-5+deb11u2) ...
Setting up libnss-systemd:amd64 (247.3-7+1-pmx11u1) ...
Setting up libatspi2.0-0:amd64 (2.38.0-4+deb11u1) ...
Setting up libzfs4linux (2.1.14-pve1) ...
Setting up grub-pc-bin (2.06-3~deb11u6) ...
Setting up librsvg2-2:amd64 (2.50.3+dfsg-1+deb11u1) ...
Setting up proxmox-kernel-helper (7.4-1) ...
Installing new version of config file /etc/grub.d/000_proxmox_boot_header ...
Installing new version of config file /etc/kernel/postinst.d/proxmox-auto-remova                                                                                 l ...
Installing new version of config file /etc/kernel/postinst.d/zz-proxmox-boot ...
Installing new version of config file /etc/kernel/postrm.d/proxmox-auto-removal                                                                                  ...
Installing new version of config file /etc/kernel/postrm.d/zz-proxmox-boot ...
Setting up grub-pc (2.06-3~deb11u6) ...
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.131-2-pve
Found initrd image: /boot/initrd.img-5.15.131-2-pve
Found linux image: /boot/vmlinuz-5.15.102-1-pve
Found initrd image: /boot/initrd.img-5.15.102-1-pve
Found linux image: /boot/vmlinuz-5.15.85-1-pve
Found initrd image: /boot/initrd.img-5.15.85-1-pve
Found linux image: /boot/vmlinuz-5.15.74-1-pve
Found initrd image: /boot/initrd.img-5.15.74-1-pve
Found memtest86+ image: /ROOT/pve-1@/boot/memtest86+.bin
Found memtest86+ multiboot image: /ROOT/pve-1@/boot/memtest86+_multiboot.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
 pve-kernel-5.15 depends on pve-kernel-5.15.131-2-pve; however:
  Package pve-kernel-5.15.131-2-pve is not configured yet.

dpkg: error processing package pve-kernel-5.15 (--configure):
 dependency problems - leaving unconfigured
Setting up librsvg2-common:amd64 (2.50.3+dfsg-1+deb11u1) ...
Setting up libpve-storage-perl (7.4-3) ...
Setting up libvirt0:amd64 (7.0.0-3+deb11u2) ...
Setting up at-spi2-core (2.38.0-4+deb11u1) ...
Setting up libpve-access-control (7.4.1) ...
Setting up libpam-systemd:amd64 (247.3-7+1-pmx11u1) ...
Setting up bind9-host (1:9.16.44-1~deb11u1) ...
Setting up libavahi-client3:amd64 (0.8-5+deb11u2) ...
Setting up libzpool5linux (2.1.14-pve1) ...
Setting up libpve-cluster-api-perl (7.3-3) ...
Setting up ssh (1:8.4p1-5+deb11u3) ...
Setting up zfsutils-linux (2.1.14-pve1) ...
Setting up dbus-user-session (1.12.28-0+deb11u1) ...
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-5.15; however:
  Package pve-kernel-5.15 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Setting up zfs-initramfs (2.1.14-pve1) ...
Setting up libpve-guest-common-perl (4.2-4) ...
Setting up libcups2:amd64 (2.3.3op2-3+deb11u6) ...
Setting up pve-firewall (4.3-5) ...
Setting up bind9-dnsutils (1:9.16.44-1~deb11u1) ...
Setting up zfs-zed (2.1.14-pve1) ...
Installing new version of config file /etc/zfs/zed.d/zed.rc ...
Setting up pve-container (4.4-6) ...
Setting up pve-ha-manager (3.6.1) ...
watchdog-mux.service is a disabled or a static unit, not starting it.
Setting up qemu-server (7.4-4) ...
Setting up pve-manager (7.4-17) ...
Installing new version of config file /etc/modprobe.d/pve-blacklist.conf ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libgdk-pixbuf-2.0-0:amd64 (2.42.2+dfsg-1+deb11u1) ...
Processing triggers for mailcap (3.69) ...
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.15.131-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount names                                                                                 pace..
Copying and configuring kernels on /dev/disk/by-uuid/5881-7511
        Copying kernel and creating boot-entry for 5.15.102-1-pve
        Copying kernel and creating boot-entry for 5.15.131-2-pve
Copying and configuring kernels on /dev/disk/by-uuid/5882-4B99
        Copying kernel and creating boot-entry for 5.15.102-1-pve
mkdir: cannot create directory ‘/var/tmp/espmounts/5882-4B99/EFI/proxmox/5.15.10                                                                                 2-1-pve’: Input/output error
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return co                                                                                 de 1
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned                                                                                  error exit status 1
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for libglib2.0-0:amd64 (2.66.8-1) ...
Setting up libgtk-3-0:amd64 (3.24.24-4+deb11u3) ...
Processing triggers for libc-bin (2.31-13+deb11u7) ...
Processing triggers for rsyslog (8.2102.0-2+deb11u1) ...
Setting up libgtk-3-bin (3.24.24-4+deb11u3) ...
Processing triggers for pve-ha-manager (3.6.1) ...
Errors were encountered while processing:
 pve-kernel-5.15.131-2-pve
 pve-kernel-5.15
 proxmox-ve
 initramfs-tools
Removing subscription nag from UI...
E: Sub-process /usr/bin/dpkg returned an error code (1)

Code:
root@RAMPLH1:~# apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
4 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up initramfs-tools (0.140) ...
update-initramfs: deferring update (trigger activated)
Setting up pve-kernel-5.15.131-2-pve (5.15.131-3) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.131-2-pve /boot/vmlinuz-5.15.131-2-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.131-2-pve /boot/vmlinuz-5.15.131-2-pve
update-initramfs: Generating /boot/initrd.img-5.15.131-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/5881-7511
        Copying kernel and creating boot-entry for 5.15.102-1-pve
        Copying kernel and creating boot-entry for 5.15.131-2-pve
Copying and configuring kernels on /dev/disk/by-uuid/5882-4B99
        Copying kernel and creating boot-entry for 5.15.102-1-pve
mkdir: cannot create directory ‘/var/tmp/espmounts/5882-4B99/EFI/proxmox/5.15.102-1-pve’: Input/output error
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-5.15.131-2-pve.postinst line 19.
dpkg: error processing package pve-kernel-5.15.131-2-pve (--configure):
 installed pve-kernel-5.15.131-2-pve package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of pve-kernel-5.15:
 pve-kernel-5.15 depends on pve-kernel-5.15.131-2-pve; however:
  Package pve-kernel-5.15.131-2-pve is not configured yet.

dpkg: error processing package pve-kernel-5.15 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-5.15; however:
  Package pve-kernel-5.15 is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.15.131-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/5881-7511
        Copying kernel and creating boot-entry for 5.15.102-1-pve
        Copying kernel and creating boot-entry for 5.15.131-2-pve
Copying and configuring kernels on /dev/disk/by-uuid/5882-4B99
        Copying kernel and creating boot-entry for 5.15.102-1-pve
mkdir: cannot create directory ‘/var/tmp/espmounts/5882-4B99/EFI/proxmox/5.15.102-1-pve’: Input/output error
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 pve-kernel-5.15.131-2-pve
 pve-kernel-5.15
 proxmox-ve
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)


Thank you in advance
 
rm: cannot remove '/var/tmp/espmounts/5882-4B99/EFI/proxmox/5.15.74-1-pve/initrd .img-5.15.74-1-pve': Read-only file system
mkdir: cannot create directory ‘/var/tmp/espmounts/5882-4B99/EFI/proxmox/5.15.10 2-1-pve’: Input/output error
mkdir: cannot create directory ‘/var/tmp/espmounts/5882-4B99/EFI/proxmox/5.15.102-1-pve’: Input/output error
mkdir: cannot create directory ‘/var/tmp/espmounts/5882-4B99/EFI/proxmox/5.15.102-1-pve’: Input/output error
Looks likes the ESP (the partition with UUID=5882-4B99) and/or its filesystem and/or its drive is broken. Maybe remove it from the list in /etc/kernel/proxmox-boot-uuids as a work-around?

EDIT: Make sure your system boots from another ESP (because that one won't work anymore) and investigate the drive/partition/filesystem problem afterwards.
 
Last edited:
Looks likes the ESP (the partition with UUID=5882-4B99) and/or its filesystem and/or its drive is broken. Maybe remove it from the list in /etc/kernel/proxmox-boot-uuids as a work-around?

EDIT: Make sure your system boots from another ESP (because that one won't work anymore) and investigate the drive/partition/filesystem problem afterwards.
5882-4B99 is the only ESP mount in that folder. So I dont think thats an option.
 
What folder are you talking about?
What is the output of cat /etc/kernel/proxmox-boot-uuids? There should at least also be 5881-7511 in there (and maybe more).
Sorry. The failure log mentioned /var/tmp/espmounts/5882-4B99 so thats what I assumed you were talking about (Please note im out of my depth on this portion of linux.

Output is as follows
Code:
root@RAMPLH1:/# cat /etc/kernel/proxmox-boot-uuids
5881-7511
5882-4B99
root@RAMPLH1:/#
 
Sorry. The failure log mentioned /var/tmp/espmounts/5882-4B99 so thats what I assumed you were talking about (Please note im out of my depth on this portion of linux.
That's probably because of the error that that (empty) folder is still there and I don't think it's relevant.
Output is as follows
5881-7511 5882-4B99
Looks like you installed Proxmox on a mirror (or RAID1) and you have two ESPs (EFI System Partitions). One of them is broken somehow. I hope your computer can (or maybe is) boot from the other ESP. You could remove the problematic 5882-4B99 from that list (in /etc/kernel/proxmox-boot-uuids) as a work-around.
 
That's probably because of the error that that (empty) folder is still there and I don't think it's relevant.

Looks like you installed Proxmox on a mirror (or RAID1) and you have two ESPs (EFI System Partitions). One of them is broken somehow. I hope your computer can (or maybe is) boot from the other ESP. You could remove the problematic 5882-4B99 from that list (in /etc/kernel/proxmox-boot-uuids) as a work-around.
Just wanted to say thanks for the help. I made a backup of proxmox-boot-uuids and the removed the 588-4B99 entry. Reboot worked and then I did another dist-upgrade which appwars to have correctly updated the kernel entries.
 
Just wanted to say thanks for the help. I made a backup of proxmox-boot-uuids and the removed the 588-4B99 entry. Reboot worked and then I did another dist-upgrade which appwars to have correctly updated the kernel entries.
Good to hear. But it sounds like you now only have one ESP in /etc/kernel/proxmox-boot-uuids and no more redundancy.
 
Good to hear. But it sounds like you now only have one ESP in /etc/kernel/proxmox-boot-uuids and no more redundancy.
So this looks to be a more widespread issue.

Just went to update my second host in this cluster and have the exact same issue. 3 host cluster all 3 hosts were 7.3.6 going to 7.4.17.

Code:
root@RAMPLH2:~# apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up initramfs-tools (0.140) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.15.74-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/62A9-63C4
        Copying kernel and creating boot-entry for 5.15.74-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/62AA-4F9F
        Copying kernel and creating boot-entry for 5.15.74-1-pve
cp: preserving times for '/var/tmp/espmounts/62AA-4F9F/EFI/proxmox/5.15.74-1-pve/initrd.img-5.15.74-1-pve': Read-only file system
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

One host seems like an anomaly. But 2 seems like an issue with this specific upgrade path
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!