Server Down after Upgrade

[Ammo]

After a routine update, my server failed to compile all packages. So when i rebooted the server went down.
I was hoping someone could advise me how to restore it.

Thanks in advance.


root@pve:~# tail -50 /var/log/syslog
Oct 9 16:42:28 pve systemd[1]: Starting PVE API Proxy Server...
Oct 9 16:42:28 pve systemd[1]: Starting PVE Cluster Ressource Manager Daemon...
Oct 9 16:42:28 pve pve-ha-crm[1377]: starting server
Oct 9 16:42:28 pve pve-ha-crm[1377]: status change startup => wait_for_quorum
Oct 9 16:42:28 pve systemd[1]: Started PVE Cluster Ressource Manager Daemon.
Oct 9 16:42:28 pve systemd[1]: Starting PVE Local HA Ressource Manager Daemon...
Oct 9 16:42:28 pve pveproxy[1373]: Can't locate PVE/Diskmanage.pm in @INC (you may need to install the PVE:iskmanage module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl) at /usr/share/perl5/PVE/API2/Ceph.pm line 20.
Oct 9 16:42:28 pve pveproxy[1373]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Ceph.pm line 20.
Oct 9 16:42:28 pve pveproxy[1373]: Compilation failed in require at /usr/share/perl5/PVE/API2/Nodes.pm line 41.
Oct 9 16:42:28 pve pveproxy[1373]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 41.
Oct 9 16:42:28 pve pveproxy[1373]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
Oct 9 16:42:28 pve pveproxy[1373]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Oct 9 16:42:28 pve pveproxy[1373]: Compilation failed in require at /usr/share/perl5/PVE/Service/pveproxy.pm line 15.
Oct 9 16:42:28 pve pveproxy[1373]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pveproxy.pm line 15.
Oct 9 16:42:28 pve pveproxy[1373]: Compilation failed in require at /usr/bin/pveproxy line 11.
Oct 9 16:42:28 pve pveproxy[1373]: BEGIN failed--compilation aborted at /usr/bin/pveproxy line 11.
Oct 9 16:42:28 pve systemd[1]: pveproxy.service: control process exited, code=exited status=2
Oct 9 16:42:28 pve systemd[1]: Failed to start PVE API Proxy Server.
Oct 9 16:42:28 pve systemd[1]: Unit pveproxy.service entered failed state.
Oct 9 16:42:28 pve systemd[1]: Starting PVE SPICE Proxy Server...
Oct 9 16:42:28 pve pve-ha-lrm[1383]: starting server
Oct 9 16:42:28 pve pve-ha-lrm[1383]: status change startup => wait_for_agent_lock
Oct 9 16:42:28 pve systemd[1]: Started PVE Local HA Ressource Manager Daemon.
Oct 9 16:42:29 pve spiceproxy[1381]: Can't locate PVE/Diskmanage.pm in @INC (you may need to install the PVE:iskmanage module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl) at /usr/share/perl5/PVE/API2/Ceph.pm line 20.
Oct 9 16:42:29 pve spiceproxy[1381]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Ceph.pm line 20.
Oct 9 16:42:29 pve spiceproxy[1381]: Compilation failed in require at /usr/share/perl5/PVE/API2/Nodes.pm line 41.
Oct 9 16:42:29 pve spiceproxy[1381]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 41.
Oct 9 16:42:29 pve spiceproxy[1381]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
Oct 9 16:42:29 pve spiceproxy[1381]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Oct 9 16:42:29 pve spiceproxy[1381]: Compilation failed in require at /usr/share/perl5/PVE/Service/spiceproxy.pm line 13.
Oct 9 16:42:29 pve spiceproxy[1381]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/spiceproxy.pm line 13.
Oct 9 16:42:29 pve spiceproxy[1381]: Compilation failed in require at /usr/bin/spiceproxy line 9.
Oct 9 16:42:29 pve spiceproxy[1381]: BEGIN failed--compilation aborted at /usr/bin/spiceproxy line 9.
Oct 9 16:42:29 pve systemd[1]: spiceproxy.service: control process exited, code=exited status=2
Oct 9 16:42:29 pve systemd[1]: Failed to start PVE SPICE Proxy Server.
Oct 9 16:42:29 pve systemd[1]: Unit spiceproxy.service entered failed state.
Oct 9 16:42:29 pve systemd[1]: Starting PVE VM Manager...
Oct 9 16:42:29 pve systemd[1]: Started LSB: Start or stop the Webmin server.
Oct 9 16:42:29 pve systemd[1]: pve-manager.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 9 16:42:29 pve systemd[1]: Failed to start PVE VM Manager.
Oct 9 16:42:29 pve systemd[1]: Unit pve-manager.service entered failed state.


root@pve:~# pveversion -v
proxmox-ve: not correctly installed (running kernel: 4.4.19-1-pve)
pve-manager: not correctly installed (running version: 4.3-1/e7cdc165)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.13-1-pve: 4.4.13-56
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-4.4.13-2-pve: 4.4.13-58
pve-kernel-4.4.15-1-pve: 4.4.15-60
pve-kernel-4.2.8-1-pve: 4.2.8-41
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.10-1-pve: 4.4.10-54
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-29
qemu-server: 4.0-88
pve-firmware: 1.1-9
libpve-common-perl: 4.0-73
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6.1-6
pve-container: 1.0-75
pve-firewall: 2.0-29
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.4-1
lxcfs: 2.0.3-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
zfsutils: 0.6.5.7-pve10~bpo80

root@pve:~# apt-get dist-upgrade
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Setting up pve-manager (4.3-1) ...
Job for pvedaemon.service failed. See 'systemctl status pvedaemon.service' and 'journalctl -xn' for details.
dpkg: error processing package pve-manager (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of proxmox-ve:
proxmox-ve depends on pve-manager; however:
Package pve-manager is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
pve-manager
proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)

pve
State: degraded
Jobs: 0 queued
Failed: 4 units
Since: Sun 2016-10-09 16:11:14 AEDT; 23min ago
CGroup: /
├─ 1 /sbin/init
├─1155 /sbin/cgmanager -m name=systemd
└─system.slice
├─zfs-zed.service
│ └─1158 /sbin/zed -F
├─ksmtuned.service
│ ├─1213 /bin/bash /usr/sbin/ksmtuned
│ └─3150 sleep 60
├─dbus.service
│ └─1187 /usr/bin/dbus-daemon --system --address=systemd: --nofork
├─cron.service
│ └─1353 /usr/sbin/cron -f
├─nfs-common.service
│ ├─1035 /sbin/rpc.statd
│ └─1049 /usr/sbin/rpc.idmapd
├─pve-ha-lrm.service
│ └─1380 pve-ha-lr
├─postfix.service
│ ├─1343 /usr/lib/postfix/master
│ ├─1344 pickup -l -t unix -u -c
│ └─1345 qmgr -l -t unix -u
├─webmin.service
│ └─1385 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/mi
├─open-iscsi.service
│ ├─1018 /usr/sbin/iscsid
│ └─1019 /usr/sbin/iscsid
├─pve-firewall.service
│ └─1359 pve-firewal
├─pve-cluster.service
│ └─1303 /usr/bin/pmxcfs
├─atd.service
│ └─1170 /usr/sbin/atd -f
├─systemd-journald.service
│ └─325 /lib/systemd/systemd-journald
├─pve-ha-crm.service
│ └─1376 pve-ha-cr
├─systemd-timesyncd.service
│ └─818 /lib/systemd/systemd-timesyncd
├─rrdcached.service
│ └─1201 /usr/bin/rrdcached -l unix:/var/run/rrdcached.sock -j /var
├─pvestatd.service
│ └─1367 pvestat
├─ssh.service
│ ├─1176 /usr/sbin/sshd -D
│ ├─2758 sshd: root@pts/0
│ ├─2768 -bash
│ ├─3175 systemctl status
│ └─3176 pager
├─systemd-logind.service
│ └─1180 /lib/systemd/systemd-logind

root@pve:~# journalctl -xn
-- Logs begin at Sun 2016-10-09 16:11:15 AEDT, end at Sun 2016-10-09 16:35:01 AEDT. --
Oct 09 16:34:18 pve pvedaemon[3130]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
Oct 09 16:34:18 pve pvedaemon[3130]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Oct 09 16:34:18 pve pvedaemon[3130]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Oct 09 16:34:18 pve pvedaemon[3130]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Oct 09 16:34:18 pve pvedaemon[3130]: Compilation failed in require at /usr/bin/pvedaemon line 11.
Oct 09 16:34:18 pve pvedaemon[3130]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11.
Oct 09 16:34:18 pve systemd[1]: pvedaemon.service: control process exited, code=exited status=2
Oct 09 16:34:18 pve systemd[1]: Failed to start PVE API Daemon.
-- Subject: Unit pvedaemon.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit pvedaemon.service has failed.
--
-- The result is failed.
Oct 09 16:34:18 pve systemd[1]: Unit pvedaemon.service entered failed state.
Oct 09 16:35:01 pve cron[1353]: (*system*pveupdate) RELOAD (/etc/cron.d/pveupdate)

below is my sources.list contents
deb http://ftp.at.debian.org/debian jessie main contrib
# security updates
deb http://security.debian.org jessie/updates main contrib

 

[dietmar]

Does it help if you run:

# apt-get dist-upgrade

 

[Ammo]

Hi Dietmar,

Thanks for the assistance. I have already run the dist-update. Below is the response.

root@pve:~# apt-get dist-upgrade
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Setting up pve-manager (4.3-1) ...
Job for pvedaemon.service failed. See 'systemctl status pvedaemon.service' and 'journalctl -xn' for details.
dpkg: error processing package pve-manager (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of proxmox-ve:
proxmox-ve depends on pve-manager; however:
Package pve-manager is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
pve-manager
proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)

 

[dietmar]

So what is the output of

# systemctl status pvedaemon.service

 

[Ammo]

Here is the output.

root@pve:~# systemctl status pvedaemon.service
● pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled)
Active: failed (Result: exit-code) since Mon 2016-10-10 16:09:28 AEDT; 6min ago
Process: 1364 ExecStart=/usr/bin/pvedaemon start (code=exited, status=2)

Oct 10 16:09:28 pve pvedaemon[1364]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 41.
Oct 10 16:09:28 pve pvedaemon[1364]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
Oct 10 16:09:28 pve pvedaemon[1364]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Oct 10 16:09:28 pve pvedaemon[1364]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Oct 10 16:09:28 pve pvedaemon[1364]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Oct 10 16:09:28 pve pvedaemon[1364]: Compilation failed in require at /usr/bin/pvedaemon line 11.
Oct 10 16:09:28 pve pvedaemon[1364]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11.
Oct 10 16:09:28 pve systemd[1]: pvedaemon.service: control process exited, code=exited status=2
Oct 10 16:09:28 pve systemd[1]: Failed to start PVE API Daemon.
Oct 10 16:09:28 pve systemd[1]: Unit pvedaemon.service entered failed state.
root@pve:~#

 

[dcsapak]

try

apt-get -f install

 

[fabian]

your libpve-storage-perl package is very outdated.. which PVE repository are you using? could you run "apt-get update" and show the output as well?

 

[Ammo]

Here is the apt-get update output

root@pve:~# apt-get update
Hit http://security.debian.org jessie/updates InRelease
Hit http://security.debian.org jessie/updates/main amd64 Packages
Hit http://security.debian.org jessie/updates/contrib amd64 Packages
Hit http://security.debian.org jessie/updates/contrib Translation-en
Hit http://security.debian.org jessie/updates/main Translation-en
Ign http://ftp.at.debian.org jessie InRelease
Hit http://ftp.at.debian.org jessie Release.gpg
Hit http://ftp.at.debian.org jessie Release
Hit http://ftp.at.debian.org jessie/main amd64 Packages
Hit http://ftp.at.debian.org jessie/contrib amd64 Packages
Hit http://ftp.at.debian.org jessie/contrib Translation-en
Hit http://ftp.at.debian.org jessie/main Translation-en
Reading package lists... Done

 

[Ammo]

Here is the output for apt-get -f install

root@pve:~# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up pve-manager (4.3-1) ...
Job for pvedaemon.service failed. See 'systemctl status pvedaemon.service' and 'journalctl -xn' for details.
dpkg: error processing package pve-manager (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of proxmox-ve:
proxmox-ve depends on pve-manager; however:
Package pve-manager is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
pve-manager
proxmox-ve
E: Sub-process /usr/bin/dpkg returned an error code (1)

 

[Ammo]

I resolved my issue. The root cause was that my pve package repositories were missing the no subscription line. I updated as per the wiki below

deb http://ftp.debian.org/debian jessie main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
deb http://download.proxmox.com/debian jessie pve-no-subscription

# security updates
deb http://security.debian.org jessie/updates main contrib

After updating to the above and performing the update and dist-upgrade everything is resolved.
Thank-you for your patience.