[SOLVED] Upgrade Issues (4to5)

zenny

Active Member
Jul 7, 2008
86
2
28
Hi,

Proxmox4 (installed on remote server on top of Debian Jessie) was serving well till upgrade in virtual machines started popping out errors making the lxc containers failing to boot. No zfs and ceph and hosting only lxc containers.

So I decided to upgrade first to Proxmox5 (https://pve.proxmox.com/wiki/Upgrad...he_basic_system_to_Debian_Stretch_and_PVE_5.0) and further upgrade to PM6 and then to PM7 as I didn't see any other way to upgrade from PM5 (Debian Stretch 9) to PM7 (Debian Bullseye 11) directly. Please correct me if I missed something.

The first error encountered is:

Code:
# apt dist-upgrade -y
...
Errors were encountered while processing:
 pve-kernel-4.15.18-30-pve
 linux-image-4.9.0-16-amd64
 linux-image-amd64
 fail2ban
 pve-kernel-4.15
 proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)
[Exit 100]
...

Further investigation led to:

Code:
# apt full-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be REMOVED:
  linux-image-3.16.0-11-amd64
The following packages have been kept back:
  grub-common grub-pc grub-pc-bin grub2-common libnvpair1linux libzfs2linux
  libzpool2linux
0 upgraded, 0 newly installed, 1 to remove and 7 not upgraded.
4 not fully installed or removed.
After this operation, 168 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 45537 files and directories currently installed.)
Removing linux-image-3.16.0-11-amd64 (3.16.84-1) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-3.16.0-11-amd64
/etc/kernel/postrm.d/zz-update-grub:
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.18-30-pve
/usr/bin/grub-mkrelpath: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/linux-image-3.16.0-11-amd64.postrm line 206.
dpkg: error processing package linux-image-3.16.0-11-amd64 (--remove):
 subprocess installed post-removal script returned error exit status 1
Errors were encountered while processing:
 linux-image-3.16.0-11-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
[Exit 100]

Methought it could be because of libnvpair issue as no zfs (only lvm), so tried to remove, but got:

Code:
# apt remove libnvpair1linux libzfs2linux libzpool2linux
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 grub-common : Depends: libnvpair1 but it is not going to be installed
               Depends: libzfs2 but it is not going to be installed
 grub-pc-bin : Depends: libnvpair1 but it is not going to be installed
               Depends: libzfs2 but it is not going to be installed
 grub2-common : Depends: libnvpair1 but it is not going to be installed
                Depends: libzfs2 but it is not going to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
[Exit 100]

Tried with 'apt-get -fy install' as well as `apt remove pve-cluster pve-ha-manager pve-container pve-manager libpve-storage-perl qemu-server libpve-guest-common-perl pve-firewall proxmox-ve` followed by `apt install proxmox-ve`, but without success. Network and hosts configurations are alright and working.

Code:
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.13 (stretch)
Release:        9.13
Codename:       stretch

Already rebooted to the new kernel by manually updating the new kernel in the grub.cfg:

Code:
# uname -a
Linux server2 4.15.18-30-pve #1 SMP PVE 4.15.18-58 (Fri, 12 Jun 2020 13:53:01 +0200) x86_64 GNU/Linux

Any inputs to get out of this shall be appreciated.

Cheers,
/z
 
Last edited:
Forgot to mention that the apt sources are also alright:

Code:
# grep '' /etc/apt/sources.list
deb [arch=amd64] http://cdn-fastly.deb.debian.org/debian/ stretch main contrib non-free
deb-src [arch=amd64] http://cdn-fastly.deb.debian.org/debian/ stretch main contrib non-free

deb [arch=amd64] http://cdn-fastly.deb.debian.org/debian/ stretch-updates main contrib non-free
deb-src [arch=amd64] http://cdn-fastly.deb.debian.org/debian/ stretch-updates main contrib non-free

# grep '' /etc/apt/sources.list.d/*
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
/etc/apt/sources.list.d/pve-enterprise.list:#deb https://enterprise.proxmox.com/debian stretch pve-enterprise
/etc/apt/sources.list.d/pve-enterprise.list.dpkg-dist:deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise
/etc/apt/sources.list.d/pve-install-repo.list:deb http://download.proxmox.com/debian stretch pve-no-subscription

~# apt update
Ign:1 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Hit:2 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Hit:3 http://cdn-fastly.deb.debian.org/debian stretch Release
Hit:4 http://download.proxmox.com/debian stretch InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
7 packages can be upgraded. Run 'apt list --upgradable' to see them.

# apt list --upgradable
Listing... Done
grub-common/stable 2.02-pve6 amd64 [upgradable from: 2.02-pve4]
grub-pc/stable 2.02-pve6 amd64 [upgradable from: 2.02-pve4]
grub-pc-bin/stable 2.02-pve6 amd64 [upgradable from: 2.02-pve4]
grub2-common/stable 2.02-pve6 amd64 [upgradable from: 2.02-pve4]
libnvpair1linux/stable 0.7.13-pve1~bpo2 amd64 [upgradable from: 0.6.5.9-pve15~bpo80]
libzfs2linux/stable 0.7.13-pve1~bpo2 amd64 [upgradable from: 0.6.5.9-pve15~bpo80]
libzpool2linux/stable 0.7.13-pve1~bpo2 amd64 [upgradable from: 0.6.5.9-pve15~bpo80]

Here appears to be some issues, apt appears to pull from pve6 (buster?)!

Does not seem a space issue as 20GB space left:

Code:
# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            977M     0  977M   0% /dev
tmpfs           200M   21M  179M  11% /run
/dev/vda2        81G   58G   20G  75% /
tmpfs           997M     0  997M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           997M     0  997M   0% /sys/fs/cgroup
tmpfs           200M     0  200M   0% /run/user/0

The dpkg cache is already clear:

Code:
# apt clean
root@server2:~# apt autoclean
Reading package lists... Done
Building dependency tree
Reading state information... Done

Don't think that it has anything to do with the default Debian kernels:

Code:
# dpkg -l | grep linux-image
iH  linux-image-3.16.0-11-amd64    3.16.84-1                         amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-4-amd64     3.16.43-2+deb8u5                  amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-5-amd64     3.16.51-3+deb8u1                  amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-6-amd64     3.16.57-2                         amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-7-amd64     3.16.59-1                         amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-8-amd64     3.16.64-2                         amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-9-amd64     3.16.68-2                         amd64        Linux 3.16 for 64-bit PCs
iF  linux-image-4.9.0-16-amd64     4.9.272-2                         amd64        Linux 4.9 for 64-bit PCs

Inside the /boot:

Code:
# ls -la /boot/
total 113540
drwxr-xr-x  3 root root     4096 Sep  2 23:43 ./
drwxr-xr-x 22 root root     4096 Sep  2 20:30 ../
-rw-r--r--  1 root root  4113110 Jun 12  2020 System.map-4.15.18-30-pve
-rw-r--r--  1 root root  3932780 Jul  5  2018 System.map-4.4.134-1-pve
-rw-r--r--  1 root root  3214704 Jul 19 22:08 System.map-4.9.0-16-amd64
-rw-r--r--  1 root root   217132 Jun 12  2020 config-4.15.18-30-pve
-rw-r--r--  1 root root   190209 Jul  5  2018 config-4.4.134-1-pve
-rw-r--r--  1 root root   186681 Jul 19 22:08 config-4.9.0-16-amd64
drwxr-xr-x  5 root root     4096 Sep  3 09:45 grub/
-rw-r--r--  1 root root 33227381 Sep  2 23:42 initrd.img-4.15.18-30-pve
-rw-r--r--  1 root root 26497902 Sep  2 12:44 initrd.img-4.4.134-1-pve
-rw-r--r--  1 root root 24574353 Sep  2 23:43 initrd.img-4.9.0-16-amd64
-rw-r--r--  1 root root  8546080 Jun 12  2020 vmlinuz-4.15.18-30-pve
-rw-r--r--  1 root root  7252880 Jul  5  2018 vmlinuz-4.4.134-1-pve
-rw-r--r--  1 root root  4269856 Jul 19 22:08 vmlinuz-4.9.0-16-amd64


Also tried to install proxmox-ve again (even used 'apt install proxmox-ve -fy') after removing, encountered with dependency hell:

Code:
# apt install proxmox-ve
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 proxmox-ve : Depends: pve-manager but it is not going to be installed
              Depends: qemu-server but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
[Exit 100]

When trying to remove the packages, it reports the pacakage not installed yet fail to install again:

Code:
# apt remove pve-cluster pve-ha-manager pve-container pve-manager libpve-storage-perl qemu-server libpve-guest-common-perl pve-firewall proxmox-ve
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'libpve-guest-common-perl' is not installed, so not removed
Package 'libpve-storage-perl' is not installed, so not removed
Package 'proxmox-ve' is not installed, so not removed
Package 'pve-cluster' is not installed, so not removed
Package 'pve-container' is not installed, so not removed
Package 'pve-firewall' is not installed, so not removed
Package 'pve-ha-manager' is not installed, so not removed
Package 'pve-manager' is not installed, so not removed
Package 'qemu-server' is not installed, so not removed
The following packages will be REMOVED:
  linux-image-3.16.0-11-amd64
0 upgraded, 0 newly installed, 1 to remove and 7 not upgraded.
4 not fully installed or removed.
After this operation, 168 MB disk space will be freed.

However, dpkg -l shows:

Code:
# dpkg -l | grep ii | grep pve
ii  ceph-common                    12.2.13-pve1~bpo9                 amd64        common utilities to mount and interact with a ceph storage cluster
ii  dmsetup                        2:1.02.137-pve6                   amd64        Linux Kernel Device Mapper userspace library
ii  libcephfs2                     12.2.13-pve1~bpo9                 amd64        Ceph distributed file system client library
ii  libdevmapper-event1.02.1:amd64 2:1.02.137-pve6                   amd64        Linux Kernel Device Mapper event support library
ii  libdevmapper1.02.1:amd64       2:1.02.137-pve6                   amd64        Linux Kernel Device Mapper userspace library
ii  liblvm2app2.2:amd64            2.02.168-pve6                     amd64        LVM2 application library
ii  libnvpair1                     0.6.5.9-pve15~bpo80               all          transitional package
ii  libnvpair1linux                0.6.5.9-pve15~bpo80               amd64        Solaris name-value library for Linux
ii  libpve-u2f-server-perl         1.0-2                             amd64        Perl bindings for libu2f-server
ii  librados2                      12.2.13-pve1~bpo9                 amd64        RADOS distributed object store client library
ii  libradosstriper1               12.2.13-pve1~bpo9                 amd64        RADOS striping interface
ii  librbd1                        12.2.13-pve1~bpo9                 amd64        RADOS block device client library
ii  librgw2                        12.2.13-pve1~bpo9                 amd64        RADOS Gateway client library
ii  libuutil1linux                 0.7.13-pve1~bpo2                  amd64        Solaris userland utility library for Linux
ii  libzfs2                        0.6.5.9-pve15~bpo80               all          transitional package
ii  libzfs2linux                   0.6.5.9-pve15~bpo80               amd64        OpenZFS filesystem library for Linux
ii  libzpool2linux                 0.6.5.9-pve15~bpo80               amd64        OpenZFS pool library for Linux
ii  pve-firmware                   2.0-7                             all          Binary firmware code for the pve-kernel
ii  pve-kernel-4.4.134-1-pve       4.4.134-112                       amd64        The Proxmox PVE Kernel Image
ii  pve-libspice-server1           0.14.1-2                          amd64        SPICE remote display system server library
ii  pve-qemu-kvm                   3.0.1-4                           amd64        Full virtualization on x86 hardware
ii  python-cephfs                  12.2.13-pve1~bpo9                 amd64        Python 2 libraries for the Ceph libcephfs library
ii  python-rados                   12.2.13-pve1~bpo9                 amd64        Python 2 libraries for the Ceph librados library
ii  python-rbd                     12.2.13-pve1~bpo9                 amd64        Python 2 libraries for the Ceph librbd library
ii  python-rgw                     12.2.13-pve1~bpo9                 amd64        Python 2 libraries for the Ceph librgw library
ii  tar                            1.29b-2+pve.1                     amd64        GNU version of the tar archiving utility

Hope, this will help figure out the issue. Thanks in advance.
 
Last edited:
Seems like a bit of a messed setup and from only the commands you posted it seems like you try quite some dangerous things...
Most of Proxmox VE got already uninstalled, when did that happen?

Also tried to install proxmox-ve again (even used 'apt install proxmox-ve -fy') after removing, encountered with dependency hell:
The following packages have unmet dependencies:
proxmox-ve:
Depends: pve-manager but it is not going to be installed
Depends: qemu-server but it is not going to be installed
That goes in the right direction, but the remedy is not adding force flags but to check for why that packages are not installed.
Try apt install <package> for each package that it lists as "is not going to be installed" (e.g., apt install pve-manager) and see for why that is, iterate and/or post the output if unsure... Sooner or later a more specific reason should be given..

In such limbo situations it sometimes can then help to pass all packages explicitly to apt install.
 
Seems like a bit of a messed setup and from only the commands you posted it seems like you try quite some dangerous things...
Most of Proxmox VE got already uninstalled, when did that happen?
Not exactly I know of. However, what I can guess is when trying to reinstall the proxmox-ve after an removing as I stated earlier.
That goes in the right direction, but the remedy is not adding force flags but to check for why that packages are not installed.
Try apt install <package> for each package that it lists as "is not going to be installed" (e.g., apt install pve-manager) and see for why that is, iterate and/or post the output if unsure... Sooner or later a more specific reason should be given..

Thanks for your input. Here comes the outputs:

Code:
# apt install pve-manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 pve-manager : Depends: libpve-access-control (>= 5.1-10) but it is not going to be installed
               Depends: libpve-guest-common-perl (>= 2.0-14) but it is not going to be installed
               Depends: libpve-storage-perl (>= 5.0-35) but it is not going to be installed
               Depends: librados2-perl but it is not going to be installed
               Depends: pve-cluster (>= 5.0-27) but it is not going to be installed
               Depends: pve-container (>= 2.0-21) but it is not going to be installed
               Depends: pve-firewall but it is not going to be installed
               Depends: pve-ha-manager but it is not going to be installed
               Depends: qemu-server (>= 5.0-24) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
[Exit 100]

# apt install qemu-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 qemu-server : Depends: libpve-access-control (>= 5.0-7) but it is not going to be installed
               Depends: libpve-guest-common-perl (>= 2.0-20) but it is not going to be installed
               Depends: libpve-storage-perl (>= 5.0-18) but it is not going to be installed
               Depends: pve-cluster but it is not going to be installed
               Depends: pve-firewall but it is not going to be installed
               Depends: pve-ha-manager but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
[Exit 100]

So this the dependency hell with broken packages appears to be the bottleneck! Andy input to overcome this would appreciated.

In such limbo situations it sometimes can then help to pass all packages explicitly to apt install.
Tried with 'apt -fy install' too, but get:

Code:
# apt -fy install
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  linux-image-3.16.0-11-amd64
0 upgraded, 0 newly installed, 1 to remove and 7 not upgraded.
4 not fully installed or removed.
After this operation, 168 MB disk space will be freed.
(Reading database ... 45537 files and directories currently installed.)
Removing linux-image-3.16.0-11-amd64 (3.16.84-1) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-3.16.0-11-amd64
/etc/kernel/postrm.d/zz-update-grub:
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.18-30-pve
/usr/bin/grub-mkrelpath: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/linux-image-3.16.0-11-amd64.postrm line 206.
dpkg: error processing package linux-image-3.16.0-11-amd64 (--remove):
 subprocess installed post-removal script returned error exit status 1
Errors were encountered while processing:
 linux-image-3.16.0-11-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
[Exit 100]


Meanwhile, to fix 'broken packages', I tried with dpkg --configure, yet no go:

Code:
root@server2:~# apt-get update --fix-missing
Ign:1 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Hit:2 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Hit:3 http://cdn-fastly.deb.debian.org/debian stretch Release
Hit:4 http://download.proxmox.com/debian stretch InRelease
Reading package lists... Done
root@server2:~# apt-get install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  linux-image-3.16.0-11-amd64
0 upgraded, 0 newly installed, 1 to remove and 7 not upgraded.
4 not fully installed or removed.
After this operation, 168 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 45537 files and directories currently installed.)
Removing linux-image-3.16.0-11-amd64 (3.16.84-1) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-3.16.0-11-amd64
/etc/kernel/postrm.d/zz-update-grub:
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.18-30-pve
/usr/bin/grub-mkrelpath: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/linux-image-3.16.0-11-amd64.postrm line 206.
dpkg: error processing package linux-image-3.16.0-11-amd64 (--remove):
 subprocess installed post-removal script returned error exit status 1
Errors were encountered while processing:
 linux-image-3.16.0-11-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
[Exit 100]

root@server2:~# dpkg --configure -a
Setting up pve-kernel-4.15.18-30-pve (4.15.18-58) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.15.18-30-pve /boot/vmlinuz-4.15.18-30-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.15.18-30-pve /boot/vmlinuz-4.15.18-30-pve
update-initramfs: Generating /boot/initrd.img-4.15.18-30-pve
W: initramfs-tools configuration sets RESUME=UUID=9bc8d7d2-7e45-43cc-b9b8-da30b5d34f46
W: but no matching swap device is available.
I: The initramfs will attempt to resume from /dev/vda1
I: (UUID=13a6beff-fc75-4dfc-bf3e-1e21d7b19b94)
I: Set the RESUME variable to override this.
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 4.15.18-30-pve /boot/vmlinuz-4.15.18-30-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.15.18-30-pve /boot/vmlinuz-4.15.18-30-pve
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.18-30-pve
/usr/bin/grub-mkrelpath: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
run-parts: /etc/kernel/postinst.d/zz-update-grub exited with return code 127
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/pve-kernel-4.15.18-30-pve.postinst line 19.
dpkg: error processing package pve-kernel-4.15.18-30-pve (--configure):
 subprocess installed post-installation script returned error exit status 2
Setting up linux-image-4.9.0-16-amd64 (4.9.272-2) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.9.0-16-amd64
W: initramfs-tools configuration sets RESUME=UUID=9bc8d7d2-7e45-43cc-b9b8-da30b5d34f46
W: but no matching swap device is available.
I: The initramfs will attempt to resume from /dev/vda1
I: (UUID=13a6beff-fc75-4dfc-bf3e-1e21d7b19b94)
I: Set the RESUME variable to override this.
/etc/kernel/postinst.d/zz-update-grub:
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.18-30-pve
/usr/bin/grub-mkrelpath: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
run-parts: /etc/kernel/postinst.d/zz-update-grub exited with return code 127
dpkg: error processing package linux-image-4.9.0-16-amd64 (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of pve-kernel-4.15:
 pve-kernel-4.15 depends on pve-kernel-4.15.18-30-pve; however:
  Package pve-kernel-4.15.18-30-pve is not configured yet.

dpkg: error processing package pve-kernel-4.15 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 pve-kernel-4.15.18-30-pve
 linux-image-4.9.0-16-amd64
 pve-kernel-4.15
[Exit 1]
# dpkg --configure pve-manager
dpkg: error processing package pve-manager (--configure):
 package pve-manager is not ready for configuration
 cannot configure (current status 'config-files')
Errors were encountered while processing:
 pve-manager
[Exit 1]

Neither it lets install anything nor remove! Else I would have first remove the entire pve-related packages and then upgrade the distro from debian stretch to buster and then to the bullseye and then reinstall proxmox-ve again.

Any clue appreciated.
 
Last edited:
I further audit the dpkg:

Code:
# dpkg --audit
The following packages have been unpacked but not yet configured.
They must be configured using dpkg --configure or the configure
menu option in dselect for them to work:
 pve-kernel-4.15      Latest Proxmox VE Kernel Image

The following packages are only half configured, probably due to problems
configuring them the first time.  The configuration should be retried using
dpkg --configure <package> or the configure menu option in dselect:
 linux-image-4.9.0-16-amd64 Linux 4.9 for 64-bit PCs
 pve-kernel-4.15.18-30-pve The Proxmox PVE Kernel Image

The following packages are only half installed, due to problems during
installation.  The installation can probably be completed by retrying it;
the packages can be removed using dselect or dpkg --remove:
 linux-image-3.16.0-11-amd64 Linux 3.16 for 64-bit PCs

Accordingly I tried first to remove the v3.16 kernel without the same libnvpair related issue, meaning has it anything to do with different versions of the grub in pve5 than pve4?

Code:
# dpkg --remove linux-image-3.16.0-11-amd64
(Reading database ... 45537 files and directories currently installed.)
Removing linux-image-3.16.0-11-amd64 (3.16.84-1) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-3.16.0-11-amd64
/etc/kernel/postrm.d/zz-update-grub:
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
/usr/sbin/grub-probe: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.18-30-pve
/usr/bin/grub-mkrelpath: symbol lookup error: /lib/libnvpair.so.1: undefined symbol: xdr_control
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/linux-image-3.16.0-11-amd64.postrm line 206.
dpkg: error processing package linux-image-3.16.0-11-amd64 (--remove):
 subprocess installed post-removal script returned error exit status 1
Errors were encountered while processing:
 linux-image-3.16.0-11-amd64
[Exit 1]

Likewise,

Code:
# dpkg --configure pve-kernel-4.15
dpkg: dependency problems prevent configuration of pve-kernel-4.15:
 pve-kernel-4.15 depends on pve-kernel-4.15.18-30-pve; however:
  Package pve-kernel-4.15.18-30-pve is not configured yet.

dpkg: error processing package pve-kernel-4.15 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 pve-kernel-4.15
[Exit 1]

Any inputs appreciated. Thanks!
 
Hi again @t.lamprecht

I removed the problem-causing packages from 'rm /var/lib/dpkg/info/<packagename>' (see at the bottom) and ran 'dpkg --configure -a' followed by 'apt update'. Some of the previous issues are gone, fyi. I have also removed all 'linux-image' packages, too. Still dependency hell of 'proxmox-ve' as well as upgrading grub-related packages are in limbo. Any clue appreciated. Thanks in advance.

Code:
# apt isntall proxmox-ve
E: Invalid operation isntall
[Exit 100]
root@server2:~# apt install proxmox-ve
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 proxmox-ve : Depends: pve-manager but it is not going to be installed
              Depends: qemu-server but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
[Exit 100]

# uname -a
Linux server2 4.4.134-1-pve #1 SMP PVE 4.4.134-112 (Thu, 05 Jul 2018 12:39:16 +0000) x86_64 GNU/Linux

# apt install proxmox-ve
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 proxmox-ve : Depends: pve-manager but it is not going to be installed
              Depends: qemu-server but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
[Exit 100]

# apt install pve-manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 pve-manager : Depends: libpve-access-control (>= 5.1-10) but it is not going to be installed
               Depends: libpve-guest-common-perl (>= 2.0-14) but it is not going to be installed
               Depends: libpve-storage-perl (>= 5.0-35) but it is not going to be installed
               Depends: librados2-perl but it is not going to be installed
               Depends: pve-cluster (>= 5.0-27) but it is not going to be installed
               Depends: pve-container (>= 2.0-21) but it is not going to be installed
               Depends: pve-firewall but it is not going to be installed
               Depends: pve-ha-manager but it is not going to be installed
               Depends: qemu-server (>= 5.0-24) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
[Exit 100]

# apt install qemu-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 qemu-server : Depends: libpve-access-control (>= 5.0-7) but it is not going to be installed
               Depends: libpve-guest-common-perl (>= 2.0-20) but it is not going to be installed
               Depends: libpve-storage-perl (>= 5.0-18) but it is not going to be installed
               Depends: pve-cluster but it is not going to be installed
               Depends: pve-firewall but it is not going to be installed
               Depends: pve-ha-manager but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
[Exit 100]

# apt list --upgradable
Listing... Done
grub-common/stable 2.02-pve6 amd64 [upgradable from: 2.02-pve4]
grub-pc/stable 2.02-pve6 amd64 [upgradable from: 2.02-pve4]
grub-pc-bin/stable 2.02-pve6 amd64 [upgradable from: 2.02-pve4]
grub2-common/stable 2.02-pve6 amd64 [upgradable from: 2.02-pve4]
libnvpair1linux/stable 0.7.13-pve1~bpo2 amd64 [upgradable from: 0.6.5.9-pve15~bpo80]
libzfs2linux/stable 0.7.13-pve1~bpo2 amd64 [upgradable from: 0.6.5.9-pve15~bpo80]
libzpool2linux/stable 0.7.13-pve1~bpo2 amd64 [upgradable from: 0.6.5.9-pve15~bpo80]

# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  grub-common grub-pc grub-pc-bin grub2-common libnvpair1linux libzfs2linux
  libzpool2linux
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.

Removed list of the packages from the '/var/lib/dpkg/info/' folder:

Code:
grub-common.conffiles                  linux-image-3.16.0-5-amd64.postrm*
grub-common.list                       linux-image-3.16.0-6-amd64.list
grub-common.md5sums                    linux-image-3.16.0-6-amd64.postrm*
grub-common.postinst*                  linux-image-3.16.0-7-amd64.list
grub-common.postrm*                    linux-image-3.16.0-7-amd64.postrm*
grub-common.preinst*                   linux-image-3.16.0-8-amd64.list
grub-common.prerm*                     linux-image-3.16.0-8-amd64.postrm*
grub-pc-bin.list                       linux-image-3.16.0-9-amd64.list
grub-pc-bin.md5sums                    linux-image-3.16.0-9-amd64.postrm*
grub-pc.conffiles                      linux-image-4.9.0-16-amd64.list
grub-pc.config*                        linux-image-4.9.0-16-amd64.md5sums
grub-pc.list                           linux-image-4.9.0-16-amd64.postinst*
grub-pc.md5sums                        linux-image-4.9.0-16-amd64.postrm*
grub-pc.postinst*                      linux-image-4.9.0-16-amd64.preinst*
grub-pc.postrm*                        linux-image-4.9.0-16-amd64.prerm*
grub-pc.preinst*                       pve-cluster.list
grub-pc.prerm*                         pve-cluster.postrm*
grub-pc.templates                      pve-container.list
grub2-common.list                      pve-container.postrm*
grub2-common.md5sums                   pve-firewall.list
libnvpair1.list                        pve-firewall.postrm*
libnvpair1.md5sums                     pve-firmware.list
libnvpair1linux.list                   pve-firmware.md5sums
libnvpair1linux.md5sums                pve-ha-manager.list
libnvpair1linux.postinst*              pve-ha-manager.postrm*
libnvpair1linux.postrm*                pve-kernel-4.15.18-30-pve.list
libnvpair1linux.shlibs                 pve-kernel-4.15.18-30-pve.md5sums
libpve-u2f-server-perl.list            pve-kernel-4.15.18-30-pve.postinst*
libpve-u2f-server-perl.md5sums         pve-kernel-4.15.18-30-pve.postrm*
librados2.list                         pve-kernel-4.15.18-30-pve.prerm*
librados2.md5sums                      pve-kernel-4.15.list
librados2.shlibs                       pve-kernel-4.15.postrm*
librados2.triggers                     pve-kernel-4.4.134-1-pve.list
libradosstriper1.list                  pve-kernel-4.4.134-1-pve.md5sums
libradosstriper1.md5sums               pve-kernel-4.4.134-1-pve.postinst*
libradosstriper1.shlibs                pve-kernel-4.4.134-1-pve.postrm*
libradosstriper1.triggers              pve-kernel-4.4.134-1-pve.prerm*
libzfs2.list                           pve-kernel-4.4.35-2-pve.list
libzfs2.md5sums                        pve-kernel-4.4.35-2-pve.postrm*
libzfs2linux.list                      pve-kernel-4.4.40-1-pve.list
libzfs2linux.md5sums                   pve-kernel-4.4.40-1-pve.postrm*
libzfs2linux.postinst*                 pve-kernel-4.4.62-1-pve.list
libzfs2linux.postrm*                   pve-kernel-4.4.62-1-pve.postrm*
libzfs2linux.shlibs                    pve-kernel-4.4.67-1-pve.list
libzpool2linux.list                    pve-kernel-4.4.67-1-pve.postrm*
libzpool2linux.md5sums                 pve-kernel-4.4.83-1-pve.list
libzpool2linux.postinst*               pve-kernel-4.4.83-1-pve.postrm*
libzpool2linux.postrm*                 pve-libspice-server1.list
libzpool2linux.shlibs                  pve-libspice-server1.md5sums
linux-image-3.16.0-11-amd64.list       pve-libspice-server1.shlibs
linux-image-3.16.0-11-amd64.md5sums    pve-libspice-server1.triggers
linux-image-3.16.0-11-amd64.postinst*  pve-manager.list
linux-image-3.16.0-11-amd64.postrm*    pve-manager.postrm*
linux-image-3.16.0-11-amd64.preinst*   pve-qemu-kvm.conffiles
linux-image-3.16.0-11-amd64.prerm*     pve-qemu-kvm.list
linux-image-3.16.0-11-amd64.templates  pve-qemu-kvm.md5sums
linux-image-3.16.0-4-amd64.list        pve-qemu-kvm.postinst*
linux-image-3.16.0-4-amd64.postrm*     qemu-server.list
linux-image-3.16.0-5-amd64.list        qemu-server.postrm*

Even tried with without success (ref:https://blog.stevedong.com/post/proxmox-ve-how-to-fixed-upgrade-error/) :

Code:
# dpkg --configure pve-manager
dpkg: error processing package pve-manager (--configure):
 package pve-manager is not ready for configuration
 cannot configure (current status 'config-files')
Errors were encountered while processing:
 pve-manager
[Exit 1]

# apt remove pve-manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'pve-manager' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.

# apt remove qemu-manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package qemu-manager
[Exit 100]

# dpkg --configure pve-manager
dpkg: error processing package pve-manager (--configure):
 package pve-manager is not ready for configuration
 cannot configure (current status 'config-files')
Errors were encountered while processing:
 pve-manager
[Exit 1]

# dpkg --configure qemu-manager
dpkg: error processing package qemu-manager (--configure):
 no package named 'qemu-manager' is installed, cannot configure
Errors were encountered while processing:
 qemu-manager
[Exit 1]

# dpkg --configure -a
root@server2:~#  proxmox-ve : Depends: pve-manager but it is not going to be installed^C
[Exit 130 (SIGINT)]

# apt remove pve-cluster pve-ha-manager pve-container pve-manager libpve-storage-perl qemu-server libpve-guest-common-perl pve-firewall proxmox-ve
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'libpve-guest-common-perl' is not installed, so not removed
Package 'libpve-storage-perl' is not installed, so not removed
Package 'proxmox-ve' is not installed, so not removed
Package 'pve-cluster' is not installed, so not removed
Package 'pve-container' is not installed, so not removed
Package 'pve-firewall' is not installed, so not removed
Package 'pve-ha-manager' is not installed, so not removed
Package 'pve-manager' is not installed, so not removed
Package 'qemu-server' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.

# apt install proxmox-ve
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 proxmox-ve : Depends: pve-manager but it is not going to be installed
              Depends: qemu-server but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
[Exit 100]

Where is the bottleneck?
 
Last edited:
SOLVED:

Posting this to make others in the same situation maybe of help:

1. What I did was, I ran `dpkg --audit` and with every problematic package, I removed (actually moved to a different location as a backup) from `/var/lib/dpkg/<packagename>`:

Code:
 mv /var/lib/dpkg/info/<problematic-packagename> /tmp
1814   │ dpkg --configure -a
1815   │ apt update
1816   │ apt full-upgrade

2. Whenever apt reports the 'holding broken packages' I got the list with:

Code:
dpkg --get-selections | grep hold | wc -l

followed by 1 above.

3. Whenever I got "warning: files list file for package '" warnings, I ran:

Code:
for package in $(apt-get upgrade 2>&1 | grep "warning: files list file for package '" | grep -Po "[^'\n ]+'" | grep -Po "[^']+"); do apt-get install --reinstall "$package"; done

Finally, I updated to buster and at the end, I installed proxmox-ve again with 'apt install proxmox-ve -y` to install pve-6 and 'apt install pve-kernel-5.4 -y' to install the v5 LTS kernel.

Hope it would save others time in the future! Thanks!
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!