[SOLVED] pve-manager update/install stuck

T

triplus

New Member
Nov 22, 2023
3
1
1
Hi

I tried doing a regular update on my single-node Proxmox homelab this morning (via GUI). The update got stuck during the "setting up" of pve-manager. I've tried doing some general troubleshooting (installing previous version, restarting, re-installing), but it keeps getting stuck on the below:

Code: 
root@proxmox:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up pve-manager (8.1.4) ...
  LVM configuration valid.

When I manage to kill the update process (it's really stuck, the only way to get it to stop is to restart the host), it shows below error:
Code: 
Job for pvescheduler.service canceled.
dpkg: error processing package pve-manager (--configure):
 installed pve-manager package post-installation script subprocess returned error exit status 1

I tried rolling back to 8.1.3, but same place where it gets stuck. I let it "hang" for 1h+ on that spot already but nothing happened. Below some troubleshooting info:

Code: 
root@proxmox:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.11-8-pve)
pve-manager: not correctly installed (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-8
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.3-1
proxmox-backup-file-restore: 3.1.3-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-3
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.2-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1


Code: 
root@proxmox:~# cat /var/log/apt/history.log

Start-Date: 2024-02-02  10:04:41
Commandline: apt-get dist-upgrade
Install: proxmox-kernel-6.5.11-8-pve-signed:amd64 (6.5.11-8, automatic)
Upgrade: pve-edk2-firmware-ovmf:amd64 (4.2023.08-2, 4.2023.08-3), libpve-rs-perl:amd64 (0.8.7, 0.8.8), proxmox-mail-forward:amd64 (0.2.2, 0.2.3), pve-edk2-firmware-legacy:amd64 (4.2023.08-2, 4.2023.08-3), proxmox-kernel-6.5:amd64 (6.5.11-7, 6.5.11-8), proxmox-backup-file-restore:amd64 (3.1.2-1, 3.1.3-1), libc6:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), locales:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), ifupdown2:amd64 (3.2.0-1+pmx7, 3.2.0-1+pmx8), pve-i18n:amd64 (3.1.5, 3.2.0), proxmox-backup-client:amd64 (3.1.2-1, 3.1.3-1), pve-manager:amd64 (8.1.3, 8.1.4), libc-l10n:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), libc-bin:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), pve-edk2-firmware:amd64 (4.2023.08-2, 4.2023.08-3)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2024-02-02  10:53:48

Start-Date: 2024-02-02  10:55:48
Commandline: apt-get dist-upgrade
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2024-02-02  11:26:12

Start-Date: 2024-02-02  11:27:40
Commandline: apt autoremove
Remove: proxmox-kernel-6.2.16-19-pve:amd64 (6.2.16-19), proxmox-kernel-6.5.11-6-pve-signed:amd64 (6.5.11-6)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2024-02-02  11:30:59

Start-Date: 2024-02-02  11:36:58
Commandline: apt install pve-manager pve-manager=8.1.3
Downgrade: pve-manager:amd64 (8.1.4, 8.1.3)

Start-Date: 2024-02-02  11:46:45
Commandline: apt install pve-manager=8.1.4
Upgrade: pve-manager:amd64 (8.1.3, 8.1.4)

Start-Date: 2024-02-02  12:22:22
Commandline: apt dist-upgrade

I tried to apt remove pve-manager, but it prompted to also remove proxmox-ve, which I wasn't comfortable with, without further guidance
 
Last edited:
I've run into this issue as well whilst trying to upgrade my Proxmox instance to the latest package versions. It appears to be hanging on deb-system-invoke reload-or-restart pvescheduler.service that is run as part of the pve-manager package post-install script. I have tried running dpkg --configure -a manually and this hangs in the same place as well.
 
Yes exactly, it seems pve-manager just doesn't want to "install" and it hangs on something. Let's hope we can get some support :)
 
I fixed my upgrade issue by downgrading all of the upgraded packages per the apt history, rebooting, then re-running apt dist-upgrade. Here are all of the steps that I followed:
  1. Downgrade all of the upgraded packages to the previously installed version using apt install <package>=<version> (the old versions on my system are listed below). Kill the stuck systemctl reload-or-restart pvescheduler.service process to allow the downgrade to finish
  2. Bulk stopped all running VMs and CTs, then ran systemctl mask pve-guests.service to stop guests from starting at boot
  3. Restart the Proxmox system
  4. Complete the pve-manager post-install script via dpkg --configure -a
  5. Re-run apt dist-upgrade
  6. Re-enabled guests to start at boot: systemctl unmask pve-guests.service then reboot again
I initially tried to downgrade specific packages, but none of the downgrades enabled the pve-manager post-install script to complete. So I'm still none the wiser about which package caused the pvescheduler systemd unit to hang on restart, but at least I have a working and upgraded system.

Code: 
Start-Date: 2024-02-04  09:55:04
Commandline: apt-get dist-upgrade
Upgrade: pve-edk2-firmware-ovmf:amd64 (4.2023.08-2, 4.2023.08-3), pve-qemu-kvm:amd64 (8.1.2-4, 8.1.2-6), pve-edk2-firmware-legacy:amd64 (4.2023.08-2, 4.2023.08-3), libproxmox-rs-perl:amd64 (0.3.1, 0.3.3), libc6:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), locales:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), ifupdown2:amd64 (3.2.0-1+pmx7, 3.2.0-1+pmx8), pve-xtermjs:amd64 (5.3.0-2, 5.3.0-3), pve-i18n:amd64 (3.1.4, 3.1.5), pve-manager:amd64 (8.1.3, 8.1.4), libc-l10n:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), libc-bin:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), pve-edk2-firmware:amd64 (4.2023.08-2, 4.2023.08-3)
 
  • Like
Reactions: xmaka
Fun fact- I was about to do as you described.

I got the error

Code: 
root@proxmox:~# apt install pve-edk2-firmware-ovmf:amd64=4.2023.08-2
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.

Since something was still wrong from last time

So I ran dpkg --configure -a expecting the system to get stuck again. But it just went trough immediately...

Looking at the installed versions, it no longer shows up as pve-manager: not correctly installed (running version: 8.1.4/ec5affc9e41f1d79)

Code: 
root@proxmox:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.11-8-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-8
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.3-1
proxmox-backup-file-restore: 3.1.3-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-3
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.2-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1

It seems for me that "waiting" was what I needed to do. The only thing I did in parallel was upgrade my proxmox backup server too (running on different hardware), but I doubt that is related.

So it seems this is resloved. Your solution is probably what is required, but for me trying again 24-48h later was the solution :)

Thanks!
 
  • Like
Reactions: i_am_jam
crowbarz said:
I fixed my upgrade issue by downgrading all of the upgraded packages per the apt history, rebooting, then re-running apt dist-upgrade. Here are all of the steps that I followed:
  1. Downgrade all of the upgraded packages to the previously installed version using apt install <package>=<version> (the old versions on my system are listed below). Kill the stuck systemctl reload-or-restart pvescheduler.service process to allow the downgrade to finish
  2. Bulk stopped all running VMs and CTs, then ran systemctl mask pve-guests.service to stop guests from starting at boot
  3. Restart the Proxmox system
  4. Complete the pve-manager post-install script via dpkg --configure -a
  5. Re-run apt dist-upgrade
  6. Re-enabled guests to start at boot: systemctl unmask pve-guests.service then reboot again
I initially tried to downgrade specific packages, but none of the downgrades enabled the pve-manager post-install script to complete. So I'm still none the wiser about which package caused the pvescheduler systemd unit to hang on restart, but at least I have a working and upgraded system.

Code: 
Start-Date: 2024-02-04  09:55:04
Commandline: apt-get dist-upgrade
Upgrade: pve-edk2-firmware-ovmf:amd64 (4.2023.08-2, 4.2023.08-3), pve-qemu-kvm:amd64 (8.1.2-4, 8.1.2-6), pve-edk2-firmware-legacy:amd64 (4.2023.08-2, 4.2023.08-3), libproxmox-rs-perl:amd64 (0.3.1, 0.3.3), libc6:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), locales:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), ifupdown2:amd64 (3.2.0-1+pmx7, 3.2.0-1+pmx8), pve-xtermjs:amd64 (5.3.0-2, 5.3.0-3), pve-i18n:amd64 (3.1.4, 3.1.5), pve-manager:amd64 (8.1.3, 8.1.4), libc-l10n:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), libc-bin:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), pve-edk2-firmware:amd64 (4.2023.08-2, 4.2023.08-3)
Click to expand...
I followed these instructions and got my system to upgrade the packages. However, pve-scheduler.service becomes dead shortly after a reboot. Upon attempting to install pve-manager again with --reinstall... it halts again...

I think something is seriously wrong with a package. I hope the proxmox devs get to looking at this soon as this is really bad.
 
crowbarz said:
I fixed my upgrade issue by downgrading all of the upgraded packages per the apt history, rebooting, then re-running apt dist-upgrade. Here are all of the steps that I followed:
  1. Downgrade all of the upgraded packages to the previously installed version using apt install <package>=<version> (the old versions on my system are listed below). Kill the stuck systemctl reload-or-restart pvescheduler.service process to allow the downgrade to finish
  2. Bulk stopped all running VMs and CTs, then ran systemctl mask pve-guests.service to stop guests from starting at boot
  3. Restart the Proxmox system
  4. Complete the pve-manager post-install script via dpkg --configure -a
  5. Re-run apt dist-upgrade
  6. Re-enabled guests to start at boot: systemctl unmask pve-guests.service then reboot again
I initially tried to downgrade specific packages, but none of the downgrades enabled the pve-manager post-install script to complete. So I'm still none the wiser about which package caused the pvescheduler systemd unit to hang on restart, but at least I have a working and upgraded system.

Code: 
Start-Date: 2024-02-04  09:55:04
Commandline: apt-get dist-upgrade
Upgrade: pve-edk2-firmware-ovmf:amd64 (4.2023.08-2, 4.2023.08-3), pve-qemu-kvm:amd64 (8.1.2-4, 8.1.2-6), pve-edk2-firmware-legacy:amd64 (4.2023.08-2, 4.2023.08-3), libproxmox-rs-perl:amd64 (0.3.1, 0.3.3), libc6:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), locales:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), ifupdown2:amd64 (3.2.0-1+pmx7, 3.2.0-1+pmx8), pve-xtermjs:amd64 (5.3.0-2, 5.3.0-3), pve-i18n:amd64 (3.1.4, 3.1.5), pve-manager:amd64 (8.1.3, 8.1.4), libc-l10n:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), libc-bin:amd64 (2.36-9+deb12u3, 2.36-9+deb12u4), pve-edk2-firmware:amd64 (4.2023.08-2, 4.2023.08-3)
Click to expand...
Thank you, that burst the knot
 
You must log in or register to reply here.
Top Bottom