Failed Proxmox 9 Upgrade From v8.4.16

Apr 30, 2024
7
0
1
Detroit, MI
History:
Attempted an in-place upgrade from v8.4.16 to v9.1 after running "pve8to9 --full" and getting no show stopping errors. At some point during the update, apt broke and left my host in an unusable state. Unfortunately, I wasn't smart enough to backup the host because I was rushing and now I'm paying the idiot tax. I'm hoping to recover this install, but if it can't be done, I'll take my lumps like a nerd and do a clean install as all VMs are on different storage from the OS.

Output of upgrade-dist:
Bash:
apt upgrade dist

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

You might want to run 'apt --fix-broken install' to correct these.

The following packages have unmet dependencies:

 dirmngr : Depends: libgnutls30t64 (>= 3.8.1) but it is not installed

 gnutls-bin : Depends: libgnutls30t64 (>= 3.8.6-0+private+1) but it is not installed

 libc-dev-bin : Depends: libc6 (< 2.37) but 2.41-12+deb13u1 is installed

 libc6-dev : Depends: libc6 (= 2.36-9+deb12u13) but 2.41-12+deb13u1 is installed

 libcurl3t64-gnutls : Depends: libgnutls30t64 (>= 3.8.6) but it is not installed

 libgnutls-dane0t64 : Depends: libgnutls30t64 (= 3.8.9-3+deb13u1) but it is not installed

 libnbd0 : Depends: libgnutls30t64 (>= 3.8.6) but it is not installed

 libngtcp2-crypto-gnutls8 : Depends: libgnutls30t64 (>= 3.8.2) but it is not installed

 pve-qemu-kvm : Depends: libgnutls30t64 (>= 3.8.6) but it is not installed

                Recommends: numactl but it is not installed

 python3 : PreDepends: python3-minimal (= 3.11.2-1+b1) but 3.13.5-1 is installed

           Depends: libpython3-stdlib (= 3.11.2-1+b1) but 3.13.5-1 is installed

 python3-bcrypt : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed

 python3-cephfs : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed

 python3-cffi-backend : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed

 python3-rados : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed

 python3-rbd : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed

 python3-rgw : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed

 samba-libs : Depends: libgnutls30t64 (>= 3.8.2) but it is not installed

 smbclient : Depends: libgnutls30t64 (>= 3.7.0) but it is not installed

 vncterm : Depends: libgnutls30t64 (>= 3.8.1) but it is not installed

E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution)

Output of apt --fix-broken install:
Bash:
apt --fix-broken install

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

Correcting dependencies... Done

The following packages were automatically installed and are no longer required:

  libabsl20220623 libaom3 libavif15 libavif16 libc-devtools libdav1d6 libdav1d7

  libde265-0 libdeflate0 libgav1-1 libgd3 libheif1 libimagequant0 libjbig0 liblerc4

  libltdl7 libnsl-dev librav1e0 librav1e0.7 libsharpyuv0 libsvtav1enc1 libsvtav1enc2

  libtiff6 libtirpc-dev libwebp7 libx265-199 libxpm4 libyuv0

  proxmox-headers-6.8.12-10-pve proxmox-headers-6.8.12-11-pve

  proxmox-headers-6.8.12-13-pve proxmox-headers-6.8.12-15-pve

  proxmox-headers-6.8.12-16-pve proxmox-headers-6.8.12-5-pve

  proxmox-headers-6.8.12-7-pve proxmox-headers-6.8.12-8-pve

  proxmox-headers-6.8.12-9-pve proxmox-kernel-6.8.12-10-pve-signed

  proxmox-kernel-6.8.12-11-pve-signed proxmox-kernel-6.8.12-13-pve-signed

  proxmox-kernel-6.8.12-15-pve-signed proxmox-kernel-6.8.12-5-pve-signed

  proxmox-kernel-6.8.12-7-pve-signed proxmox-kernel-6.8.12-8-pve-signed

  proxmox-kernel-6.8.12-9-pve-signed python3-attr python3-distro python3-docker

  python3-dockerpty python3-docopt python3-dotenv python3-json-pointer

  python3-jsonschema python3-pyrsistent python3-rfc3987 python3-texttable

  python3-uritemplate python3-webcolors python3-websocket

Use 'apt autoremove' to remove them.

The following additional packages will be installed:

  docker-compose libc-dev-bin libc6-dev libgnutls30t64 libhogweed6t64 libp11-kit0

  libprotobuf32t64 lxc-pve python3 python3-apt python3-protobuf python3-pycurl

  python3-pyrsistent python3-systemd python3-yaml

Suggested packages:

  libc-devtools glibc-doc python3-doc python3-tk python3-venv python-apt-doc

  libcurl4-gnutls-dev python-pycurl-doc

Recommended packages:

  docker-cli

The following packages will be REMOVED:

  libgnutls30 libgnutlsxx30 libhogweed6 libprotobuf32 python3-distutils

  python3-lib2to3

The following NEW packages will be installed:

  libgnutls30t64 libhogweed6t64 libprotobuf32t64

The following packages will be upgraded:

  docker-compose libc-dev-bin libc6-dev libp11-kit0 lxc-pve python3 python3-apt

  python3-protobuf python3-pycurl python3-pyrsistent python3-systemd python3-yaml

12 upgraded, 3 newly installed, 6 to remove and 502 not upgraded.

254 not fully installed or removed.

Need to get 0 B/20.1 MB of archives.

After this operation, 63.3 MB of additional disk space will be used.

Do you want to continue? [Y/n] Y

Traceback (most recent call last):

  File "/usr/bin/apt-listchanges", line 29, in <module>

    import apt_pkg

ModuleNotFoundError: No module named 'apt_pkg'

(Reading database ... 424551 files and directories currently installed.)

Preparing to unpack .../docker-compose_2.26.1-4_amd64.deb ...

Unpacking docker-compose (2.26.1-4) over (1.29.2-3) ...

dpkg: error processing archive /var/cache/apt/archives/docker-compose_2.26.1-4_amd64.deb (--unpack):

 trying to overwrite '/usr/libexec/docker/cli-plugins/docker-compose', which is also in package docker-compose-plugin 5.0.1-1~debian.12~bookworm

dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)

Errors were encountered while processing:

 /var/cache/apt/archives/docker-compose_2.26.1-4_amd64.deb

E: Sub-process /usr/bin/dpkg returned an error code (1)

Output of pveversion -v:
Bash:
proxmox-ve: 8.4.0 (running kernel: 6.8.12-17-pve)

pve-manager: not correctly installed (running version: 8.4.16/368e3c45c15b895c)

proxmox-kernel-6.8: 6.8.12-17

proxmox-kernel-6.8.12-17-pve-signed: 6.8.12-17

proxmox-kernel-6.8.12-16-pve-signed: 6.8.12-16

proxmox-kernel-6.8.12-15-pve-signed: 6.8.12-15

proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13

proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11

proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10

proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9

proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8

proxmox-kernel-6.8.12-7-pve-signed: 6.8.12-7

proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5

proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4

ceph-fuse: not correctly installed

corosync: 3.1.9-pve1

criu: 3.17.1-2+deb12u2

glusterfs-client: 10.3-5

ifupdown2: 3.2.0-1+pmx11

intel-microcode: 3.20250211.1~deb12u1

ksm-control-daemon: 1.5-1

libjs-extjs: 7.0.0-5

libknet1: 1.30-pve2

libproxmox-acme-perl: not correctly installed

libproxmox-backup-qemu0: not correctly installed

libproxmox-rs-perl: not correctly installed

libpve-access-control: not correctly installed

libpve-apiclient-perl: 3.3.2

libpve-cluster-api-perl: not correctly installed

libpve-cluster-perl: not correctly installed

libpve-common-perl: not correctly installed

libpve-guest-common-perl: not correctly installed

libpve-http-server-perl: not correctly installed

libpve-network-perl: 0.11.3

libpve-rs-perl: not correctly installed

libpve-storage-perl: not correctly installed

libspice-server1: not correctly installed

lvm2: 2.03.16-2

lxc-pve: 6.0.0-2

lxcfs: 6.0.0-pve2

novnc-pve: 1.6.0-2

proxmox-backup-client: 3.4.7-1

proxmox-backup-file-restore: 3.4.7-1

proxmox-backup-restore-image: 0.7.0

proxmox-firewall: 0.7.1

proxmox-kernel-helper: not correctly installed

proxmox-mail-forward: 0.3.3

proxmox-mini-journalreader: 1.5

proxmox-offline-mirror-helper: 0.6.8

proxmox-widget-toolkit: 4.3.13

pve-cluster: not correctly installed

pve-container: 5.3.3

pve-docs: 8.4.1

pve-edk2-firmware: 4.2025.02-4~bpo12+1

pve-esxi-import-tools: 0.7.4

pve-firewall: not correctly installed

pve-firmware: 3.16-3

pve-ha-manager: not correctly installed

pve-i18n: 3.4.5

pve-qemu-kvm: not correctly installed

pve-xtermjs: 5.5.0-2

qemu-server: not correctly installed

smartmontools: 7.3-pve1

spiceterm: 3.3.1

swtpm: 0.8.0+pve1

vncterm: not correctly installed

zfsutils-linux: 2.2.8-pve1

I've also attached the output of /var/log/apt/history.log just in case.
 

Attachments

Last edited:
Hi.
1) Please show the result of apt-get update (of course using the < / > button).
2) What are the package sources?
 
Updated the original post using code blocks as requested and here are the requested outputs:

Bash:
apt-get update
Hit:1 http://deb.debian.org/debian trixie InRelease
Hit:2 http://security.debian.org/debian-security trixie-security InRelease
Hit:3 http://deb.debian.org/debian trixie-updates InRelease
Hit:4 http://repo.netdata.cloud/repos/stable/debian trixie/ InRelease
Hit:5 http://repository.netdata.cloud/repos/stable/debian trixie/ InRelease
Hit:6 http://repo.netdata.cloud/repos/repoconfig/debian trixie/ InRelease
Hit:7 http://repository.netdata.cloud/repos/repoconfig/debian trixie/ InRelease
Hit:8 https://packages.cloud.google.com/apt coral-edgetpu-stable InRelease
Hit:9 http://download.proxmox.com/debian/pve trixie InRelease
Reading package lists... Done
W: https://packages.cloud.google.com/apt/dists/coral-edgetpu-stable/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

Bash:
apt policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
     release o=Proxmox,a=stable,n=trixie,l=Proxmox Debian Repository,c=pve-no-subscription,b=amd64
     origin download.proxmox.com
 500 http://repository.netdata.cloud/repos/repoconfig/debian trixie/ Packages
     release v=13,o=Netdata,n=trixie,l=Netdata,c=
     origin repository.netdata.cloud
 500 http://repository.netdata.cloud/repos/stable/debian trixie/ Packages
     release v=13,o=Netdata,n=trixie,l=Netdata,c=
     origin repository.netdata.cloud
 500 http://repo.netdata.cloud/repos/repoconfig/debian trixie/ Packages
     release v=13,o=Netdata,n=trixie,l=Netdata,c=
     origin repo.netdata.cloud
 500 http://repo.netdata.cloud/repos/stable/debian trixie/ Packages
     release v=13,o=Netdata,n=trixie,l=Netdata,c=
     origin repo.netdata.cloud
 500 https://packages.cloud.google.com/apt coral-edgetpu-stable/main all Packages
     release o=coral-edgetpu-stable,a=coral-edgetpu-stable,n=coral-edgetpu-stable,l=coral-edgetpu-stable,c=main,b=all
     origin packages.cloud.google.com
 500 https://packages.cloud.google.com/apt coral-edgetpu-stable/main amd64 Packages
     release o=coral-edgetpu-stable,a=coral-edgetpu-stable,n=coral-edgetpu-stable,l=coral-edgetpu-stable,c=main,b=amd64
     origin packages.cloud.google.com
 500 http://security.debian.org/debian-security trixie-security/main amd64 Packages
     release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=main,b=amd64
     origin security.debian.org
 500 http://deb.debian.org/debian trixie-updates/main amd64 Packages
     release v=13-updates,o=Debian,a=stable-updates,n=trixie-updates,l=Debian,c=main,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/contrib amd64 Packages
     release v=13.3,o=Debian,a=stable,n=trixie,l=Debian,c=contrib,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/main amd64 Packages
     release v=13.3,o=Debian,a=stable,n=trixie,l=Debian,c=main,b=amd64
     origin deb.debian.org
 
Hi,
Code:
Unpacking docker-compose (2.26.1-4) over (1.29.2-3) ...

dpkg: error processing archive /var/cache/apt/archives/docker-compose_2.26.1-4_amd64.deb (--unpack):

 trying to overwrite '/usr/libexec/docker/cli-plugins/docker-compose', which is also in package docker-compose-plugin 5.0.1-1~debian.12~bookworm
sounds like you might need to remove either the docker-compose or docker-compose-plugin package. Since the docker-compose-plugin has a version for Debian 12, I'd try to remove that, e.g. apt --fix-broken install docker-compose-plugin-
 
I think you got bit by the difference between: apt-get upgradeand apt-get dist-upgrade.

You really want to use apt-get dist-upgrade.Try using the correct command and see if you can recover the upgrade.
 
Thanks, but unfortunately, that made no difference.

Bash:
apt upgrade-dist
E: Invalid operation upgrade-dist
root@ard-host1:/# apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 dirmngr : Depends: libgnutls30t64 (>= 3.8.1) but it is not installed
 gnutls-bin : Depends: libgnutls30t64 (>= 3.8.6-0+private+1) but it is not installed
 libc-dev-bin : Depends: libc6 (< 2.37) but 2.41-12+deb13u1 is installed
 libc6-dev : Depends: libc6 (= 2.36-9+deb12u13) but 2.41-12+deb13u1 is installed
 libcurl3t64-gnutls : Depends: libgnutls30t64 (>= 3.8.6) but it is not installed
 libgnutls-dane0t64 : Depends: libgnutls30t64 (= 3.8.9-3+deb13u1) but it is not installed
 libnbd0 : Depends: libgnutls30t64 (>= 3.8.6) but it is not installed
 libngtcp2-crypto-gnutls8 : Depends: libgnutls30t64 (>= 3.8.2) but it is not installed
 pve-qemu-kvm : Depends: libgnutls30t64 (>= 3.8.6) but it is not installed
                Recommends: numactl but it is not installed
 python3 : PreDepends: python3-minimal (= 3.11.2-1+b1) but 3.13.5-1 is installed
           Depends: libpython3-stdlib (= 3.11.2-1+b1) but 3.13.5-1 is installed
 python3-bcrypt : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed
 python3-cephfs : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed
 python3-cffi-backend : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed
 python3-rados : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed
 python3-rbd : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed
 python3-rgw : Depends: python3 (>= 3.13~) but 3.11.2-1+b1 is installed
 samba-libs : Depends: libgnutls30t64 (>= 3.8.2) but it is not installed
 smbclient : Depends: libgnutls30t64 (>= 3.7.0) but it is not installed
 vncterm : Depends: libgnutls30t64 (>= 3.8.1) but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
root@host:/# apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libabsl20220623 libaom3 libavif15 libavif16 libc-devtools libdav1d6 libdav1d7 libde265-0 libdeflate0 libgav1-1 libgd3 libheif1
  libimagequant0 libjbig0 liblerc4 libltdl7 libnsl-dev librav1e0 librav1e0.7 libsharpyuv0 libsvtav1enc1 libsvtav1enc2 libtiff6
  libtirpc-dev libwebp7 libx265-199 libxpm4 libyuv0 proxmox-headers-6.8.12-10-pve proxmox-headers-6.8.12-11-pve
  proxmox-headers-6.8.12-13-pve proxmox-headers-6.8.12-15-pve proxmox-headers-6.8.12-16-pve proxmox-headers-6.8.12-5-pve
  proxmox-headers-6.8.12-7-pve proxmox-headers-6.8.12-8-pve proxmox-headers-6.8.12-9-pve proxmox-kernel-6.8.12-10-pve-signed
  proxmox-kernel-6.8.12-11-pve-signed proxmox-kernel-6.8.12-13-pve-signed proxmox-kernel-6.8.12-15-pve-signed
  proxmox-kernel-6.8.12-5-pve-signed proxmox-kernel-6.8.12-7-pve-signed proxmox-kernel-6.8.12-8-pve-signed
  proxmox-kernel-6.8.12-9-pve-signed python3-attr python3-distro python3-docker python3-dockerpty python3-docopt python3-dotenv
  python3-json-pointer python3-jsonschema python3-pyrsistent python3-rfc3987 python3-texttable python3-uritemplate python3-webcolors
  python3-websocket
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  docker-compose libc-dev-bin libc6-dev libgnutls30t64 libhogweed6t64 libp11-kit0 libprotobuf32t64 lxc-pve python3 python3-apt
  python3-protobuf python3-pycurl python3-pyrsistent python3-systemd python3-yaml
Suggested packages:
  libc-devtools glibc-doc python3-doc python3-tk python3-venv python-apt-doc libcurl4-gnutls-dev python-pycurl-doc
Recommended packages:
  docker-cli
The following packages will be REMOVED:
  libgnutls30 libgnutlsxx30 libhogweed6 libprotobuf32 python3-distutils python3-lib2to3
The following NEW packages will be installed:
  libgnutls30t64 libhogweed6t64 libprotobuf32t64
The following packages will be upgraded:
  docker-compose libc-dev-bin libc6-dev libp11-kit0 lxc-pve python3 python3-apt python3-protobuf python3-pycurl python3-pyrsistent
  python3-systemd python3-yaml
12 upgraded, 3 newly installed, 6 to remove and 502 not upgraded.
254 not fully installed or removed.
Need to get 0 B/20.1 MB of archives.
After this operation, 63.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Traceback (most recent call last):
  File "/usr/bin/apt-listchanges", line 29, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
(Reading database ... 424551 files and directories currently installed.)
Preparing to unpack .../docker-compose_2.26.1-4_amd64.deb ...
Unpacking docker-compose (2.26.1-4) over (1.29.2-3) ...
dpkg: error processing archive /var/cache/apt/archives/docker-compose_2.26.1-4_amd64.deb (--unpack):
 trying to overwrite '/usr/libexec/docker/cli-plugins/docker-compose', which is also in package docker-compose-plugin 5.0.1-1~debian.12~bookworm
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/docker-compose_2.26.1-4_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
I resolved the ModuleNotFoundError: No module named 'apt_pkg'
error by running the following commands:
Bash:
cd /usr/lib/python3/dist-packages
cp apt_pkg.cpython-311-x86_64-linux-gnu.so apt_pkg.so

Running apt remove docker-compose-plugin- didn't resolve the docker related error however.
 
It looks like I got the install back on track.

dpkg -i --force-overwrite /usr/libexec/docker/cli-plugins/docker-compose

Followed by apt -f install, a poorly advised reboot, and my networking config breaking again (bridge wouldn't start and system acted like it was ignoring the config file which forced me to set IP and routes manually). At that point I was able to run apt dist-upgrade twice, with an apt update in-between which seems to have resolved everything.