Server keeps booting into emergency mode

[sigh]

Full disclosure: I'm running an old version of Proxmox 3.4 which I have been planning to update to 4.x but haven't got around to yet. The server is colocated at a datacenter in the Netherlands and I have access via an iDRAC (Dell IPMI system). I have very limited linux knowledge, so please forgive my ignorance.

The server runs a number of VMs, all of which are KVM, mainly for the purposes of hosting websites. It was running perfectly fine until earlier today when I received a system email:

wget (1.13.4-3+deb7u3) wheezy-security; urgency=high

  This version fixes a security vulnerability (CVE-2016-4971) present
  in all old versions of wget.  The vulnerability was discovered by
  Dawid Golunski which were reported to us by Beyond Security's
  SecuriTeam.

  On a server redirect from HTTP to a FTP resource, wget would trust the
  HTTP server and uses the name in the redirected URL as the destination
  filename.
  This behaviour was changed and now it works similarly as a redirect
  from HTTP to another HTTP resource so the original name is used as
  the destination file.  To keep the previous behaviour the user must
  provide --trust-server-names.

 -- Thorsten Alteholz <debian@alteholz.de>  Mon, 27 Jun 2016 18:00:14 +0200

I'm used to getting such emails when I update the system myself, but I hadn't logged in recently. The server was unresponsive to ping, so I had the datacenter reboot the system. No change.

I used the iDRAC to reboot the system myself so I could watch the startup sequence and was eventually presented with the message below.



I entered the root password and tried to follow what the prompt said. I ran systemctrl default and got the following message.

Failed to get D-Bus connection: No connection to service manager

When I ran journalctl I got the message "command not found".

Has anyone got any ideas?

At this stage, since I have no idea what's wrong, all I want to do is see if I can get the vm images off and onto another server. Do you think I could do this using a live cd and if so, how would I go about it?

I would appreciate any help people could provide.

 

[dcsapak]

mhmm it seems somehow systemd got installed?
maybe someone ran an upgrade to 4.x which got interrupted?
do you have any logs under /var/log/apt which would indicate an update?
what do the files /etc/apt/sources.list and /etc/apt/sources.list.d/* say?

 

[sigh]

Thanks for the quick response. Sorry it takes me so long to reply. I'm a lawyer during the day so I only get the chance to look at this at night.

The history.log for apt shows that the following commands were executed:

apt-get -y install git
apt-get install systemd-sysv
apt-get -yg install make gcc lipcre3-dev libssl-dev wget
apt-get install -y --no-install-recommends python python-dev

term.log suggests that sysvinit was removed. It ends after setting up python-virtualenv.

The sources.list file is as follows (and it doesn't look good):

# security updates
deb security.deian.org/ wheezy/updates main contrib
deb deb.torproject.org/torproject.org wheezy main
deb download.proxmox.com/debian wheexy pve-no-subscription

I had to remove http etc because of posting restrictions.

The reference to tor seems to indicate the system has been compromised, since I didn't put it there.

The only file in sources.list.d is pve-enterprise.list and the only line in that is commented out because I don't have a subscription.

I'm wondering if maybe if maybe it might be easier to just get the datacenter to make and insert a Debian live cd. I should then be able to mount all the relevant folders following the instructions from the wiki page: Recover_From_Grub_Failure, copy the VMs off and then restart afresh with proxmox 4.4. Alternatively, I'm open to any other suggestions you have.

Thanks in advance!

 

[dcsapak]

The reference to tor seems to indicate the system has been compromised, since I didn't put it there.

if you think your system is compromised, i would urge you to wipe the system and restore from backup
you can never know what an attacker left on your system, how he manipulated the logs, etc.