pvedaemon & pveproxy down after system restart

[f.schaaf2]

Hi.

This weekend, after a system reset, the services pvedaemon & pveproxy did not start anymore.
the log mentions perl5 could it have to do something with it?
In order to restore office service, we started every important vm manually. But the webui is down and pvenodes are unable to automatically start etc.

pveversion:

root@pve3:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: not correctly installed (running version: 7.1-11/8d529482)
pve-kernel-helper: 7.1-13
pve-kernel-5.13: 7.1-6
pve-kernel-5.4: 6.4-12
pve-kernel-5.3: 6.1-6
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.4.162-1-pve: 5.4.162-2
pve-kernel-4.15: 5.4-9
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-7
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-6
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

pveproxy log:

Mar 27 17:15:13 pve3 systemd[1]: Starting PVE API Proxy Server...
Mar 27 17:15:14 pve3 pveproxy[4286]: Can't locate Proxmox/RS/CalendarEvent.pm in @INC (you may need to install the Proxmox::RS::CalendarEvent module) (@INC contains: /e>Mar 27 17:15:14 pve3 pveproxy[4286]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Backup.pm line 19.
Mar 27 17:15:14 pve3 pveproxy[4286]: Compilation failed in require at /usr/share/perl5/PVE/API2/Cluster.pm line 25.
Mar 27 17:15:14 pve3 pveproxy[4286]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 25.
Mar 27 17:15:14 pve3 pveproxy[4286]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
Mar 27 17:15:14 pve3 pveproxy[4286]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Mar 27 17:15:14 pve3 pveproxy[4286]: Compilation failed in require at /usr/share/perl5/PVE/Service/pveproxy.pm line 16.
Mar 27 17:15:14 pve3 pveproxy[4286]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pveproxy.pm line 16.
Mar 27 17:15:14 pve3 pveproxy[4286]: Compilation failed in require at /usr/bin/pveproxy line 11.
Mar 27 17:15:14 pve3 pveproxy[4286]: BEGIN failed--compilation aborted at /usr/bin/pveproxy line 11.
Mar 27 17:15:14 pve3 systemd[1]: pveproxy.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Mar 27 17:15:14 pve3 systemd[1]: pveproxy.service: Failed with result 'exit-code'.
Mar 27 17:15:14 pve3 systemd[1]: Failed to start PVE API Proxy Server.
Mar 27 17:15:14 pve3 systemd[1]: pveproxy.service: Scheduled restart job, restart counter is at 3.
Mar 27 17:15:14 pve3 systemd[1]: Stopped PVE API Proxy Server.

pvedaemon:

Mar 27 17:15:08 pve3 systemd[1]: Starting PVE API Daemon...
Mar 27 17:15:09 pve3 pvedaemon[4234]: Can't locate Proxmox/RS/CalendarEvent.pm in @INC (you may need to install the Proxmox::RS::CalendarEvent module) (@INC contains: /e>
Mar 27 17:15:09 pve3 pvedaemon[4234]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Backup.pm line 19.
Mar 27 17:15:09 pve3 pvedaemon[4234]: Compilation failed in require at /usr/share/perl5/PVE/API2/Cluster.pm line 25.
Mar 27 17:15:09 pve3 pvedaemon[4234]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 25.
Mar 27 17:15:09 pve3 pvedaemon[4234]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
Mar 27 17:15:09 pve3 pvedaemon[4234]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Mar 27 17:15:09 pve3 pvedaemon[4234]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Mar 27 17:15:09 pve3 pvedaemon[4234]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Mar 27 17:15:09 pve3 pvedaemon[4234]: Compilation failed in require at /usr/bin/pvedaemon line 11.
Mar 27 17:15:09 pve3 pvedaemon[4234]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11.
Mar 27 17:15:09 pve3 systemd[1]: pvedaemon.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Mar 27 17:15:09 pve3 systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Mar 27 17:15:09 pve3 systemd[1]: Failed to start PVE API Daemon.
Mar 27 17:15:09 pve3 systemd[1]: pvedaemon.service: Scheduled restart job, restart counter is at 1.
Mar 27 17:15:09 pve3 systemd[1]: Stopped PVE API Daemon.

 

[fabian]

can you try running apt update; apt full-upgrade and post the output here?

 

[f.schaaf2]

Thank you so much. We will try that as soon as operational safety allows.

 

[simplysoft]

We ran into the exact same issue, package installation of pve-manager failed.
Using apt full-upgrade solved the problem.

Setting up pve-manager (7.1-11) ...
Job for pvedaemon.service failed.
See "systemctl status pvedaemon.service" and "journalctl -xe" for details.
dpkg: error processing package pve-manager (--configure):
 installed pve-manager package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 pve-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)

 

[ronsrussell]

FYI -
Same issue here after upgrade to 7.2-3.
We were also prompted to run apt --fix-broken install
All is good now.