6.2 to 6.4 update pvedaemon failure with pve-manager held back

[pragmaticpro]

In preparation of wanting to upgrade to version 7, I read in the upgrade notes that it was preferable to update to 6.4 before proceeding. Unfortunately, something has gone wrong during the update/upgrade from 6.2 to 6.4 and a few packages are being held back. Now I'm no longer able to access the GUI.

Reviewing terminal history, I ran:
apt update
apt upgrade
apt dist-upgrade

It was at this point, that I was prompted with the warning of upgrading from V6 to V7 and upon reviewing the upgrade documentation, decided to cancel the upgrade when presented with the option to do so.

I modified /etc/apt/sources.list to comment out the bullseye releases and added the buster releases (expected that this would allow me to force the update to V6.4).

root@pve:~# cat /etc/apt/sources.list
#deb http://ftp.debian.org/debian bullseye main contrib
#deb http://ftp.debian.org/debian bullseye-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
#deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# security updates
#deb http://security.debian.org/debian-security bullseye-security main contrib

deb http://ftp.debian.org/debian buster main contrib
deb http://ftp.debian.org/debian buster-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

# security updates
deb http://security.debian.org/debian-security buster/updates main contrib

Re-running the update/upgrade/dist-upgrade processes did install a number of packages, but a few still remain and are what I suspect is causing pve to no longer work properly:

root@pve:~# apt update
Hit:1 http://security.debian.org/debian-security buster/updates InRelease
Hit:2 http://ftp.debian.org/debian buster InRelease
Hit:3 http://ftp.debian.org/debian buster-updates InRelease
Hit:4 http://download.proxmox.com/debian/pve buster InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
5 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@pve:~# apt list --upgradable
Listing... Done
libpve-common-perl/stable 6.4-5 all [upgradable from: 6.1-2]
libpve-storage-perl/stable 6.4-1 all [upgradable from: 6.1-7]
pve-container/stable 3.3-6 all [upgradable from: 3.1-5]
pve-manager/stable 6.4-15 amd64 [upgradable from: 6.2-4]
qemu-server/stable 6.4-2 amd64 [upgradable from: 6.2-2]

root@pve:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  libpve-common-perl libpve-storage-perl pve-container pve-manager qemu-server
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

Attempting to install the held back packages manually:

root@pve:~# apt install qemu-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 qemu-server : Depends: libpve-common-perl (>= 6.3-3) but 6.1-2 is to be installed
               Depends: libpve-storage-perl (>= 6.3-8) but 6.1-7 is to be installed
E: Unable to correct problems, you have held broken packages.

Here's pveversion -v:

root@pve:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.10.0-14-amd64)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.4-20
pve-kernel-helper: 6.4-20
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-5
libpve-storage-perl: 6.1-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 7.2-1
pve-container: 3.1-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.4-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.2-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.7-pve1

apt-mark showhold shows nothing:

apt-mark showhold

Here's the response when checking on pvedaemon:

systemctl status pvedaemon.service

pvedaemon.service - PVE API Daemon
   Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2023-08-31 22:12:03 EDT; 13h ago
  Process: 1118 ExecStart=/usr/bin/pvedaemon start (code=exited, status=255/EXCEPTION)

Aug 31 22:12:03 pve systemd[1]: Failed to start PVE API Daemon.
Aug 31 22:12:03 pve systemd[1]: pvedaemon.service: Start request repeated too quickly.
Aug 31 22:12:03 pve systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Aug 31 22:12:03 pve systemd[1]: Failed to start PVE API Daemon.
Aug 31 22:12:04 pve systemd[1]: pvedaemon.service: Start request repeated too quickly.
Aug 31 22:12:04 pve systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Aug 31 22:12:04 pve systemd[1]: Failed to start PVE API Daemon.
Aug 31 22:12:05 pve systemd[1]: pvedaemon.service: Start request repeated too quickly.
Aug 31 22:12:05 pve systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Aug 31 22:12:05 pve systemd[1]: Failed to start PVE API Daemon.
root@pve:~#  systemctl start pvedaemon.service
Job for pvedaemon.service failed because the control process exited with error code.
See "systemctl status pvedaemon.service" and "journalctl -xe" for details.

journalctl -xe didn't return much that's helpful:

Sep 01 11:20:03 pve systemd[1]: pvedaemon.service: Control process exited, code=exited, status=255/EXCEPTION
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit pvedaemon.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 255.
Sep 01 11:20:03 pve systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pvedaemon.service has entered the 'failed' state with result 'exit-code'.
Sep 01 11:20:03 pve systemd[1]: Failed to start PVE API Daemon.
-- Subject: A start job for unit pvedaemon.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pvedaemon.service has finished with a failure.
--
-- The job identifier is 67527 and the job result is failed.
Sep 01 11:20:04 pve systemd[1]: pvedaemon.service: Service RestartSec=100ms expired, scheduling restart.
Sep 01 11:20:04 pve systemd[1]: pvedaemon.service: Scheduled restart job, restart counter is at 5.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Automatic restarting of the unit pvedaemon.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Sep 01 11:20:04 pve systemd[1]: Stopped PVE API Daemon.
-- Subject: A stop job for unit pvedaemon.service has finished
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A stop job for unit pvedaemon.service has finished.
--
-- The job identifier is 67612 and the job result is done.
Sep 01 11:20:04 pve systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
-- Subject: A start job for unit corosync.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit corosync.service has finished successfully.
--
-- The job identifier is 67696.
Sep 01 11:20:04 pve systemd[1]: pvedaemon.service: Start request repeated too quickly.
Sep 01 11:20:04 pve systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pvedaemon.service has entered the 'failed' state with result 'exit-code'.
Sep 01 11:20:04 pve systemd[1]: Failed to start PVE API Daemon.
-- Subject: A start job for unit pvedaemon.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pvedaemon.service has finished with a failure.
--
-- The job identifier is 67612 and the job result is failed.

Additional debug output:

root@pve:~# apt list --installed | grep linux-image
linux-image-5.10.0-14-amd64/now 5.10.113-1 amd64 [installed,local]

root@pve:~# apt --fix-broken install
Reading package lists... 0%
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

root@pve:~# uname -a
Linux pve 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux

Current vzdump state:

root@pve:~# vzdump
no such standard option 'prune-backups'
Compilation failed in require at /usr/share/perl5/PVE/VZDump/Plugin.pm line 11.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/VZDump/Plugin.pm line 11.
Compilation failed in require at /usr/share/perl5/PVE/API2/Qemu.pm line 34.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Qemu.pm line 34.
Compilation failed in require at /usr/share/perl5/PVE/HA/Resources/PVEVM.pm line 16.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Resources/PVEVM.pm line 19.
Compilation failed in require at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 21.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 21.
Compilation failed in require at /usr/share/perl5/PVE/VZDump.pm line 18.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/VZDump.pm line 18.
Compilation failed in require at /usr/share/perl5/PVE/API2/VZDump.pm line 13.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/VZDump.pm line 13.
Compilation failed in require at /usr/share/perl5/PVE/CLI/vzdump.pm line 8.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/CLI/vzdump.pm line 8.
Compilation failed in require at /usr/bin/vzdump line 6.
BEGIN failed--compilation aborted at /usr/bin/vzdump line 6.

I'm willing to start fresh with V7 but vzdump no longer works either.
If this ends up not being fixable to the point of getting 6.4 working, my next course of action will be attempting to figure out if it's possible to backup the couple vm's I have in order to rebuild them on a fresh install of V7/V8.
Is it possible to backup the VM's without vzdump working?

 

[pragmaticpro]

Any thoughts if this is due to the current kernel version vs the version that pve-container and pve-manager are currently installed at?

 

[Neobin]

root@pve:~# uname -a
Linux pve 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux

This is the default Debian kernel, instead of the PVE one.
I personally would removing the metapackage: linux-image-amd64 like mentioned here: [1] and afterwards running a: apt full-upgrade again give a try.

apt upgrade

Never use only: apt upgrade with Proxmox products!: [2]

[1] https://pve.proxmox.com/wiki/Upgrad...rade_wants_to_remove_package_.27proxmox-ve.27
[2] https://forum.proxmox.com/threads/proxmox-ve-7-1-released.99847/post-463941

 

[pragmaticpro]

@Neobin Thanks for the info.

I tried to remove linux-image-amd64, but apt claims it's not installed...

root@pve:/etc/apt# apt remove linux-image-amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'linux-image-amd64' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

Anything else to try?

 

[Neobin]

While it is "somewhat wrong" that the default Debian kernel is booted instead of the PVE one, this might not be the (main) issue here, because I just saw this:

pve-cluster: 7.2-1

So, it seems you already partly upgraded from 6 to 7.

Unfortunately, I do not know how to get out of this state in a safe way, sorry.

 

[pragmaticpro]

@Neobin Doing a bit more research, I'm starting to suspect this is due to a perl-base version.


Attempting to explicitly install pve-cluster 6.4-1 manually, there's a missing dependency perlapi-5.28.1, which I have also seen as a missing dependency for the 5 packages above that won't upgrade.

root@pve:~# apt install pve-cluster=6.4-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 pve-cluster : Depends: perlapi-5.28.1
E: Unable to correct problems, you have held broken packages.
root@pve:~# apt install perlapi-5.28.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package perlapi-5.28.1 is a virtual package provided by:
  perl-base 5.28.1-6+deb10u1 [Not candidate version]

E: Package 'perlapi-5.28.1' has no installation candidate
root@pve:~# apt install perl-base 5.28.1-6+deb10u1
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package 5.28.1-6+deb10u1
E: Couldn't find any package by glob '5.28.1-6+deb10u1'
E: Couldn't find any package by regex '5.28.1-6+deb10u1'
root@pve:~# apt list -a perl-base
Listing... Done
perl-base/now 5.32.1-4+deb11u2 amd64 [installed,local]
perl-base/oldoldstable 5.28.1-6+deb10u1 amd64

I'm not entirely familiar with the kernel and what you were mentioning earlier with the non PVE version that I somehow have, but does this line point to the debian version being the culprit for the perl-5.28.1 incompatibility?

version '5.28.1-6+deb10u1' (Debian:10.13/oldoldstable [amd64])

root@pve:~# uname -a
Linux pve 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux

root@pve:~# apt list -a perl-base
Listing... Done
perl-base/now 5.32.1-4+deb11u2 amd64 [installed,local]
perl-base/oldoldstable 5.28.1-6+deb10u1 amd64

It looks like Debian 5.10.0 is in-use, but perl-base that's installed mentions "deb11u2".

Also, in case it's relevant:

root@pve:~# find /boot/vmli*
/boot/vmlinuz-5.10.0-14-amd64
/boot/vmlinuz-5.4.203-1-pve
/boot/vmlinuz-5.4.34-1-pve

root@pve:~# dpkg --list | grep linux-image
ii  linux-image-5.10.0-14-amd64          5.10.113-1                             amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-amd64                    5.10.113-1                             amd64        Linux for 64-bit PCs (meta-package)

root@pve:~# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
5.10.0-14-amd64
5.4.203-1-pve
5.4.34-1-pve

 

[Neobin]

Out of curiosity, what is the output of?: cat /etc/os-release

but perl-base that's installed mentions "deb11u2".

Yes, this version (5.32.1-4+deb11u2) is also already from Debian 11/Bullseye (on which is PVE 7 based on).
So this is another indicator, that the system partly (aka: at least some packages) got already upgraded to Debian Bullseye (and PVE 7).

As I said, since I am no expert at all and you even seem to have no backups, I unfortunately can not help you in a safe way in this state, sorry. (Obviously, I do not want to suggest anything that might break your system even further.)

Hopefully, maybe after the weekend, someone can help you out.

 

[pragmaticpro]

Here's os-release:

root@pve:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

 

[pragmaticpro]

Regarding backups: I have made a copy of the physical disks at this point (after the issue occurred unfortunately), so I'm not too concerned with "making things worse".

 

[Neobin]

Here's os-release:

root@pve:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Okay, still on Debian 10/Buster there, but this is not meaning much, since it was simply not yet updated at this point; but obviously does not mean that some packages not got already upgraded, like we can see by fact.

Was mainly out of curiosity anyway.

Regarding backups: I have made a copy of the physical disks at this point (after the issue occurred unfortunately), so I'm not too concerned with "making things worse".

I guess, there are two options:

1. Downgrade all already upgraded packages, which, depending on the amount, might mean going down a rabbit hole really quick.
2. Proceed with the upgrade to PVE 7 and Debian 11/Bullseye.

I personally would try to proceed with the upgrade: [1].
In case you really also decide to, please triple check all the configured repositories, especially also in the: /etc/apt/sources.list.d/ directory and I would highly recommend to start the actual upgrade (apt full-upgrade or apt dist-upgrade) through a physical connected keyboard and monitor or IPMI and not remotely (e.g.: SSH).

If you run in any warning/error messages, please additionally to the actual message(s) also provide the full output in code-tags (or good readable screenshots/pictures, if necessary) each of:

• grep -r '' /etc/apt/sources.list*
• apt update
• apt full-upgrade

Obviously, I can not guarantee you anything!

[1] https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0