Help Recovering Host

Mam89

New Member
May 31, 2018
4
3
3
34
"She's dead Jim!"

Hello, and thank you guys for an awesome KVM experience! I did something dumb a couple months back and ran apt-get upgrade (I swear, I've learned my lesson!) and borked my systems updates. Lucky for me all my VMs and Web interface kept going for some reason, yay! Except today I was attempting to find a fix and I think I finally killed the poor thing dead.

Current setup is Host OS was set up on a 2x2TB ZFS Raid1, host is still accessible via SSH (how I finally killed it heh). All my data is residing on a completely different pool than the host, thank goodness.

Some additional information before I ask for options:

Output of pveversion -v

Code:
proxmox-ve: not correctly installed (running kernel: 4.15.17-3-pve)
pve-manager: not correctly installed (running version: 5.2-3/785ba980)
pve-kernel-4.15: 5.2-3
pve-kernel-4.13: 5.1-45
pve-kernel-4.15.17-3-pve: 4.15.17-13
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-3-pve: 4.13.16-49
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.4-1-pve: 4.13.4-26
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-34
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-23
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-27
pve-container: 2.0-23
pve-docs: 5.2-4
pve-firewall: 3.0-12
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9

Removed pve-manager and rebooted

When attempting to reinstall pve-manager (apt install pve-manager):
Code:
Reading package lists... Done
Building dependency tree
Reading state information... Done
pve-manager is already the newest version (5.2-3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Setting up pve-manager (5.2-3) ...
/var/lib/dpkg/info/pve-manager.postinst: 41: /var/lib/dpkg/info/pve-manager.postinst: cannot create /root/.forward: Directory nonexistent
dpkg: error processing package pve-manager (--configure):
 subprocess installed post-installation script returned error exit status 2
Errors were encountered while processing:
 pve-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)

Tried to go through this process here: (shortened for antispam)/wiki/Install_Proxmox_VE_on_Debian_Stretch

But when get the following error when trying to input repository key:
Code:
HTTP request sent, awaiting response... 500 Internal Server Error
2018-07-16 01:40:00 ERROR 500: Internal Server Error.

apt update && apt dist-upgrade :
Code:
E: The repository 'http://download.proxmox.com/debian stretch Release' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository '(changed for spam filter)//download.proxmox.com/debian/ceph-luminous stretch Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository '(changed for spam filter)//download.proxmox.com/debian/pve stretch Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository '(changed for spam filter)//ftp.us.debian.org/debian stretch Release' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository '(changed for spam filter)//security.debian.org stretch/updates Release' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.]

And finally what I could think of adding, my /etc/apt/sources.list
Code:
deb (changed for spam filter)//ftp.us.debian.org/debian stretch main contrib

# security updates
deb (changed for spam filter)//security.debian.org stretch/updates main contrib
deb (changed for spam filter)//download.proxmox.com/debian stretch pve-no-subscription

So, I'm kinda stumped at the moment in which direction to go... It looks like I'm having several different issues which are all culminating in a non-working pve-manager.

Any assistance in the best way to fix or even recreate the node (and remount my zpool) would be awesome!

Thank you ;)
 
/var/lib/dpkg/info/pve-manager.postinst: 41: /var/lib/dpkg/info/pve-manager.postinst: cannot create /root/.forward: Directory nonexistent
seems the /root directory is missing ? how?

But when get the following error when trying to input repository key:
Code:
HTTP request sent, awaiting response... 500 Internal Server Error
2018-07-16 01:40:00 ERROR 500: Internal Server Error.
apt update && apt dist-upgrade :
Code:
E: The repository 'http://download.proxmox.com/debian stretch Release' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository '(changed for spam filter)//download.proxmox.com/debian/ceph-luminous stretch Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository '(changed for spam filter)//download.proxmox.com/debian/pve stretch Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository '(changed for spam filter)//ftp.us.debian.org/debian stretch Release' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository '(changed for spam filter)//security.debian.org stretch/updates Release' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.]
there seems to be a network issue, are you behind a proxy?
 
seems the /root directory is missing ? how?

That sir is a good question.... I have no idea where or when the /root directory went missing.

As far as the Proxy, it looks like my Sophos XG Home UTM decided to turn on a trial period of their Sandstorm DPI software, thus creating a proxy environment and messing up my key registration. As soon as I turned all the DPI features off I was able to successfully register the key and perform a apt update && apt dist-upgrade.

It looks like everything updated to the newest versions, I went ahead and rebooted the host.

And now the system is working again! And it looks like all my VMs are booting up as we speak in their proper boot order.

Just for clearity, to make sure everything went through right here is the new pveversion -v:

Code:
proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve)
pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
pve-kernel-4.15: 5.2-4
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-1-pve: 4.15.18-15
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.16-3-pve: 4.13.16-50
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.4-1-pve: 4.13.4-26
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-35
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-24
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-28
pve-container: 2.0-24
pve-docs: 5.2-4
pve-firewall: 3.0-13
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9

Hopefully that healed any damage I did and the system can start serving my plex goodness again :D
 
  • Like
Reactions: guletz
Okay so everything came up properly, except for one issue.

All my Windows Server 2016 (3 VM) and Windows 10 (3 VM) were experiencing no network connection (unable to ping anything inside or outside of the network). I did some troubleshooting and found some issues in my local DNS, made repairs there but that didn't resolve the disconnect. I thought maybe there was a driver issue, so I downloaded the newest stable drivers for VirtIO and installed those, no change. I changed the setting on on the individual VM nics from VirtIO to E1000 and all the VMs picked up DHCP and came online. Changed my servers back to static settings internally and they're fine now.

There seems to be a misconfig or bug with VirtIO and Win10/Server2016 somewhere in my system.

EDIT: To clarify, none of my linux systems (also running virtio) are having issues.... Grr Windows!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!