[SOLVED] port 8006 closed after upgrade

[Felix Pastron]

I am testing an upgrade following this guide - https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0

Everything goes well, but after a restart, I lose access to the web interface. Looks like port 8006 is closed. I have tried this 5 times now, on a few different machines, and always the same outcome. I still have SSH access, but that is it. Any input/help is much appreciated.

# pveversion -v

proxmox-ve: 5.3-1 (running kernel: 4.4.134-1-pve)
pve-manager: 5.3-11 (running version: 5.3-11/d4907f84)
pve-kernel-4.15: 5.3-3
pve-kernel-4.15.18-12-pve: 4.15.18-35
pve-kernel-4.4.134-1-pve: 4.4.134-112
pve-kernel-4.4.35-1-pve: 4.4.35-77
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-47
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-12
libpve-storage-perl: 5.0-39
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-23
pve-cluster: 5.0-33
pve-container: 2.0-35
pve-docs: 5.3-3
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-18
pve-firmware: 2.0-6
pve-ha-manager: 2.0-8
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-2
pve-xtermjs: 3.10.1-2
qemu-server: 5.0-47
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.6.5.9-pve15~bpo80

 

[oguz]

What is the output of

systemctl status pveproxy
journalctl -u pveproxy

 

[Felix Pastron]

What is the output of

systemctl status pveproxy
journalctl -u pveproxy

root@proxmox4:~# systemctl status pveproxy
Failed to get properties: Launch helper exited with unknown return code 1
root@proxmox4:~# journalctl -u pveproxy
No journal files were found.
-- No entries --

 

[oguz]

Check if you have systemd running:

ps ax | grep systemd

if there's no systemd process, try to find out why it's not running. Maybe you're not booting with it?

How exactly did you perform this update? Anything outside of what's mentioned in the guide for "In-place upgrade"?

 

[Felix Pastron]

root@proxmox4:~# ps ax | grep systemd
  595 ?        Ss     0:00 /lib/systemd/systemd-udevd --daemon
 4327 pts/0    S+     0:00 grep systemd

I downloaded the 4.4 ISO from your website, installed it. Updated it to the latest 4.4 version with apt-get update && apt-get dist-upgrade, then just followed the guide for In-place upgrade. That was it.

 

[oguz]

I downloaded the 4.4 ISO from your website, installed it. Updated it to the latest 4.4 version with apt-get update && apt-get dist-upgrade, then just followed the guide for In-place upgrade. That was it.

Is this a fresh installation? Why don't you use the latest 5.3-2 ISO instead?

If not a fresh install but an upgrade of an old host, then I really can't tell much from the information you've provided. It should work properly if you followed the steps, unless you've made some modifications on the system by yourself.

 

[Felix Pastron]

The 4.4 was a fresh install on a test machine. I am testing the upgrade process because one of our production servers is still running the 4.4 version. I would like to upgrade it to 5.3, instead of a fresh install, to minimize the downtime of the VMs on it.

I appreciate you trying to help. Is there any other info/data I can provide to figure this out? I just re-did the upgrade again last night on a different machine and have not restarted it yet, so the web interface still works and port 8006 is still open.

 

[oguz]

The 4.4 was a fresh install on a test machine. I am testing the upgrade process because one of our production servers is still running the 4.4 version. I would like to upgrade it to 5.3, instead of a fresh install, to minimize the downtime of the VMs on it.

I see.

Is there any other info/data I can provide to figure this out? I just re-did the upgrade again last night on a different machine and have not restarted it yet, so the web interface still works and port 8006 is still open.

On another thread, someone was having problems with systemd after a package upgrade. You might be having the same or a similar problem. Take a look here: https://forum.proxmox.com/threads/upgrade-to-kernel-4-15-18-12-pve-goes-boom.52564/

If this is a test machine anyway, I'd suggest you to run the workaround from Thomas before rebooting the server and see if that works:

apt purge insserv sysv-rc initscripts openrc
apt autoremove --purge
apt install --reinstall systemd

and then reboot.

If the problem persists after this, then try installing a fresh 4.4 again, following the steps on the guide. Post the output of every command that you run, but most interesting is what happens during the package upgrade itself (e.g. apt update or apt dist-upgrade, which pkgs get installed/removed or if there are any warnings).

 

[Felix Pastron]

Ok thank you, I will try that workaround today. I did save the package info from the 5.3 upgrade last night (i did not save the actual output while it was upgrading):

The following packages will be REMOVED:
  init libcwidget3 libkadm5clnt-mit9 libkadm5srv-mit9 libkdb5-7 libpython3.4-minimal libpython3.4-stdlib libsigc++-2.0-0c2a
  perl-modules python3.4 python3.4-minimal systemd-sysv sysv-rc
The following NEW packages will be installed:
  attr ceph-fuse corosync cpp-6 dirmngr distro-info-data ebtables fonts-font-awesome gcc-6-base genisoimage gnome-icon-theme
  gnupg-l10n gtk-update-icon-cache hicolor-icon-theme initramfs-tools-core iso-codes keyutils libacl1-dev libapt-inst2.0
  libapt-pkg5.0 libattr1-dev libauthen-sasl-perl libbabeltrace-ctf1 libbabeltrace1 libbind9-140 libboost-filesystem1.62.0
  libboost-iostreams1.62.0 libboost-program-options1.62.0 libboost-random1.62.0 libboost-regex1.62.0 libboost-system1.62.0
  libboost-thread1.62.0 libbytes-random-secure-perl libc-dev-bin libc-l10n libc6-dev libcfg6 libcmap4 libcorosync-common4
  libcpg4 libcroco3 libcrypt-random-seed-perl libcwidget3v5 libdns-export162 libdns162 libdw1 libefiboot1 libefivar1 libeinfo1
  libelf1 libfastjson4 libfcgi-bin libfcgi0ldbl libfdisk1 libfdt1 libgail-common libgail18 libgmime-2.6-0 libgnutls30
  libhogweed4 libicu57 libidn2-0 libintl-xs-perl libip4tc0 libip6tc0 libiptc0 libisc-export160 libisc160 libisccc140
  libisccfg140 libiscsi7 libisl15 libisns0 libjson-c3 libldap-common liblinear3 liblognorm5 liblua5.3-0 liblwres141 liblz4-1
  libmagic-mgc libmailtools-perl libmath-random-isaac-perl libmath-random-isaac-xs-perl libnet-smtp-ssl-perl libnet1
  libnetfilter-conntrack3 libnettle6 libnghttp2-14 libnl-3-200 libnl-route-3-200 libnotmuch4 libnpth0 libopus0 libperl5.24
  libpng16-16 libprocps6 libprotobuf10 libpsl5 libpve-apiclient-perl libpython3.5-minimal libpython3.5-stdlib libquorum5
  libradosstriper1 librc1 libreadline7 librgw2 librrd8 librsvg2-2 librsvg2-common libsecret-1-0 libsecret-common
  libsigc++-2.0-0v5 libsnappy1v5 libssl1.0.2 libssl1.1 libstatgrab10 libtotem-pg5 libtypes-serialiser-perl libunistring0
  liburcu4 libvotequorum8 libxapian30 libxslt1.1 libxtables12 linux-base linux-libc-dev lsb-release lynx-common manpages-dev
  openrc perl-modules-5.24 perl-openssl-defaults postfix-sqlite proxmox-widget-toolkit publicsuffix pve-edk2-firmware pve-i18n
  pve-kernel-4.15 pve-kernel-4.15.18-12-pve pve-xtermjs python-cephfs python-cffi-backend python-click python-colorama
  python-cryptography python-enum34 python-httplib2 python-idna python-ipaddress python-pyasn1 python-pycurl python-pysimplesoap
  python-rados python-rbd python-setuptools python-simplejson python3-apt python3-chardet python3-debian python3-debianbts
  python3-httplib2 python3-pkg-resources python3-pycurl python3-pysimplesoap python3-reportbug python3-requests python3-six
  python3-urllib3 python3.5 python3.5-minimal sysvinit-core tex-common xsltproc xxd
The following packages have been kept back:
  zfs-initramfs zfsutils-linux
The following packages will be upgraded:
  acl adduser apparmor apt apt-listchanges apt-transport-https apt-utils aptitude aptitude-common at base-files base-passwd bash
  bash-completion bc bind9-host bootlogd bridge-utils bsd-mailx bsdmainutils bsdutils btrfs-progs btrfs-tools busybox bzip2
  ca-certificates ceph-common cifs-utils console-setup console-setup-linux coreutils corosync-pve cpio cpp criu cron cstream
  dash dbus dc debconf debconf-i18n debian-archive-keyring debian-faq debianutils dh-python diffutils dmeventd dmidecode dmsetup
  dnsutils doc-debian docutils-common dosfstools dpkg dtach e2fslibs e2fsprogs efibootmgr eject faketime fdutils file findutils
  fontconfig fontconfig-config fonts-dejavu-core ftp fuse gawk gdisk gettext-base glusterfs-client glusterfs-common gnupg
  gnupg-agent gnupg2 gpgv grep groff-base grub-common grub-efi-amd64-bin grub-efi-ia32-bin grub-pc grub-pc-bin grub2-common gzip
  hdparm hostname ifenslave ifenslave-2.6 ifupdown info init-system-helpers initramfs-tools initscripts insserv install-info
  iproute2 ipset iptables iputils-ping isc-dhcp-client isc-dhcp-common kbd keyboard-configuration klibc-utils kmod krb5-locales
  ksm-control-daemon less libacl1 libaio1 libanyevent-http-perl libanyevent-perl libapparmor-perl libapparmor1 libappconfig-perl
  libapt-pkg-perl libarchive13 libasound2 libasound2-data libassuan0 libasyncns0 libatk1.0-0 libatk1.0-data libattr1
  libaudit-common libaudit1 libauthen-pam-perl libavahi-client3 libavahi-common-data libavahi-common3 libblas-common libblas3
  libblkid1 libbsd0 libbz2-1.0 libc-bin libc6 libcairo2 libcap-ng0 libcap2 libcap2-bin libcephfs1 libclone-perl libcloog-isl4
  libcomerr2 libcommon-sense-perl libconvert-asn1-perl libcorosync4-pve libcrypt-openssl-bignum-perl
  libcrypt-openssl-random-perl libcrypt-openssl-rsa-perl libcrypt-ssleay-perl libcryptsetup4 libcups2 libcurl3-gnutls libdatrie1
  libdb5.3 libdbi1 libdbus-1-3 libdebconfclient0 libdevmapper-event1.02.1 libdevmapper1.02.1 libdirectfb-1.2-9 libedit2
  libencode-locale-perl libestr0 libev-perl libev4 libevent-2.0-5 libexpat1 libfaketime libffi6 libfile-sync-perl
  libfilesys-df-perl libflac8 libfontconfig1 libfreetype6 libfuse2 libgc1c2 libgcc1 libgcrypt20 libgdbm3 libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-common libgeoip1 libgfortran3 libglib2.0-0 libgmp10 libgnutls-openssl27 libgnutlsxx28 libgoogle-perftools4
  libgpg-error0 libgpgme11 libgpm2 libgraphite2-3 libgssapi-krb5-2 libgssrpc4 libgtk2.0-0 libgtk2.0-common libharfbuzz0b
  libhtml-parser-perl libhtml-tagset-perl libhtml-tree-perl libhttp-message-perl libibverbs1 libice6 libidn11 libintl-perl
  libio-multiplex-perl libio-socket-inet6-perl libio-socket-ssl-perl libio-stringy-perl libipset3 libjbig0 libjemalloc1
  libjpeg62-turbo libjs-extjs libjs-jquery libjson-perl libjson-xs-perl libk5crypto3 libkeyutils1 libklibc libkmod2 libkrad0
  libkrb5-3 libkrb5support0 libksba8 libldap-2.4-2 libldb1 liblinux-inotify2-perl liblocale-gettext-perl liblockfile-bin
  liblockfile1 liblog-agent-perl liblogging-stdlog0 liblua5.2-0 liblvm2app2.2 liblvm2cmd2.02 liblzma5 liblzo2-2 libmagic1
  libmath-bigint-perl libmime-base32-perl libmnl0 libmount1 libmpc3 libmpdec2 libmpfr4 libncurses5 libncursesw5 libnet-dbus-perl
  libnet-dns-perl libnet-http-perl libnet-ldap-perl libnet-ssleay-perl libnewt0.52 libnfsidmap2 libnspr4 libnss3 libntdb1
  libnuma1 libnvpair1linux libp11-kit0 libpam-modules libpam-modules-bin libpam-runtime libpam0g libpango-1.0-0
  libpangocairo-1.0-0 libpangoft2-1.0-0 libpcap0.8 libpci3 libpcre3 libperl4-corelibs-perl libpipeline1 libpixman-1-0 libpopt0
  libprotobuf-c1 libpth20 libpulse0 libpve-access-control libpve-common-perl libpve-guest-common-perl libpve-http-server-perl
  libpve-storage-perl libpython-stdlib libpython2.7 libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libqb0
  libquadmath0 librados2 librados2-perl librbd1 librdmacm1 libreadline5 librpcsecgss3 librrds-perl librtmp1 libsasl2-2
  libsasl2-modules-db libsdl1.2debian libseccomp2 libselinux1 libsemanage-common libsemanage1 libsepol1 libsigsegv2 libslang2
  libsm6 libsmartcols1 libsmbclient libsndfile1 libsocket6-perl libsqlite3-0 libss2 libssh2-1 libstdc++6
  libstring-shellquote-perl libsystemd0 libtalloc2 libtasn1-6 libtcmalloc-minimal4 libtdb1 libtemplate-perl
  libterm-readline-gnu-perl libtevent0 libtext-charwidth-perl libtext-iconv-perl libtext-unidecode-perl libtext-wrapi18n-perl
  libthai-data libthai0 libtiff5 libtinfo5 libtirpc1 libtokyocabinet9 libudev1 libunwind8 liburi-perl libusb-0.1-4 libusb-1.0-0
  libusbredirparser1 libustr-1.0-1 libuuid-perl libuuid1 libuutil1linux libverto-libev1 libverto1 libvorbis0a libvorbisenc2
  libwbclient0 libwgdb0 libwrap0 libwww-perl libx11-6 libx11-data libx11-xcb1 libxaw7 libxcb-render0 libxcb-shm0 libxcb1
  libxcomposite1 libxcursor1 libxdamage1 libxdmcp6 libxext6 libxfixes3 libxft2 libxi6 libxinerama1 libxkbfile1
  libxml-libxml-perl libxml-parser-perl libxml-twig-perl libxml2 libxmu6 libxmuu1 libxpm4 libxrandr2 libxrender1 libxt6 libxtst6
  libzfs2linux libzpool2linux locales login logrotate lsb-base lsof lvm2 lxc-pve lxcfs lynx lynx-cur lzop m4 man-db manpages
  mawk memtest86+ mime-support mlocate mount multiarch-support mutt nano ncurses-base ncurses-bin ncurses-term net-tools netbase
  netcat-traditional nfacct nfs-common nmap novnc-pve numactl omping open-iscsi openssh-client openssh-server
  openssh-sftp-server openssl passwd patch pciutils perl perl-base pinentry-gtk2 postfix procmail procps proxmox-ve psmisc
  pve-cluster pve-container pve-docs pve-firewall pve-firmware pve-ha-manager pve-libspice-server1 pve-manager pve-qemu-kvm
  python python-apt python-apt-common python-ceph python-chardet python-debian python-debianbts python-docutils python-flask
  python-itsdangerous python-jinja2 python-markupsafe python-minimal python-pkg-resources python-protobuf python-requests
  python-roman python-six python-talloc python-urllib3 python-werkzeug python-wstools python2.7 python2.7-minimal python3
  python3-minimal qemu-server readline-common reportbug rpcbind rrdcached rsync rsyslog samba-common samba-libs sed
  sensible-utils sgml-base shared-mime-info smartmontools smbclient socat spiceterm spl sqlite3 ssh ssl-cert startpar strace
  systemd sysvinit-utils tar tasksel tasksel-data tcpdump telnet texinfo thin-provisioning-tools time traceroute tzdata ucf udev
  uidmap usbutils util-linux vim-common vim-tiny vncterm w3m wget whiptail whois x11-apps x11-common xauth xfsprogs xkb-data
  xml-core xz-utils zlib1g
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  init systemd-sysv (due to init)
530 upgraded, 175 newly installed, 13 to remove and 2 not upgraded.
Need to get 376 MB of archives.
After this operation, 870 MB of additional disk space will be used.

 

[oguz]

Looks like it's the same problem (systemd-sysv being removed). I think the workaround should solve the issue. Maybe also run a apt upgrade && apt full-upgrade afterwards just to make sure.

 

[Felix Pastron]

Looks like it's the same problem (systemd-sysv being removed). I think the workaround should solve the issue. Maybe also run a apt upgrade && apt full-upgrade afterwards just to make sure.

Ok thank you I will try that soon. Did you mean apt update? (You said apt upgrade)

 

[oguz]

Did you mean apt update? (You said apt upgrade)

Yes sorry

Running this after the workaround:

apt update
apt full-upgrade

should get everything straight.

 

[Felix Pastron]

Cool no problem, thank you

 

[Felix Pastron]

Yes sorry

Running this after the workaround:

apt update
apt full-upgrade

should get everything straight.

By the way, with Proxmox, is there a difference between apt and apt-get? The upgrade article refers to apt-get commands.

 

[oguz]

Quoting Stoiko on the other thread: https://forum.proxmox.com/threads/upgrade-to-kernel-4-15-18-12-pve-goes-boom.52564/#post-243462

For `apt` (the high-level package manager and new frontend for `apt-get`, `apt-cache`, ...) `apt full-upgrade` and `apt dist-upgrade` call the same code and can be considered synonymous.

* `apt-get` (same source-package and codebase) has `dist-upgrade` only
* `aptitude` (different codebase and a separate source-package) has `full-upgrade` only (which afaik does the same, but has e.g. a different conflict resolution mechanism)

IOW - if you use `apt` you can use both `full-upgrade` and `dist-upgrade` interchangably.

Hope this helps!

 

[Felix Pastron]

I see.



On another thread, someone was having problems with systemd after a package upgrade. You might be having the same or a similar problem. Take a look here: https://forum.proxmox.com/threads/upgrade-to-kernel-4-15-18-12-pve-goes-boom.52564/

If this is a test machine anyway, I'd suggest you to run the workaround from Thomas before rebooting the server and see if that works:

apt purge insserv sysv-rc initscripts openrc
apt autoremove --purge
apt install --reinstall systemd

and then reboot.

If the problem persists after this, then try installing a fresh 4.4 again, following the steps on the guide. Post the output of every command that you run, but most interesting is what happens during the package upgrade itself (e.g. apt update or apt dist-upgrade, which pkgs get installed/removed or if there are any warnings).

I did the steps you mentioned above, tried to reboot and got this error message:

root@proxmox4:~# init 6
-bash: init: command not found

 

[oguz]

PVE supports only systemd. Try 'systemctl reboot' or just 'reboot'.

 

[Felix Pastron]

Got it, thank you.

Looks like the system is shot after restart, can't SSH into it anymore. Errors in console:

Can't process LV pve/data: thin-pool target support missing from kernel?
run-init: /sbin/init: No such file or directory
run-init: /etc/init: Permission denied
run-init: /bin/init: No such file or directory
/bin/sh: 0: can't access tty; job control turned off

 

[oguz]

Hi,
we just tried reproducing and solving this issue. We were able to successfully do that, updating from 4.0 to the newest 5.3.
Apparently this problem appears only when the zfs packages are installed (without the zfs packages the upgrade problem didn't occur).

We're updating the wiki page now with the workaround.

EDIT:

Edited the wiki page accordingly. You can try to upgrade again from a fresh install.

 

[Felix Pastron]

Hi,
we just tried reproducing and solving this issue. We were able to successfully do that, updating from 4.0 to the newest 5.3.
Apparently this problem appears only when the zfs packages are installed (without the zfs packages the upgrade problem didn't occur).

We're updating the wiki page now with the workaround.

EDIT:

Edited the wiki page accordingly. You can try to upgrade again from a fresh install.

Oh wow, that was fast, thank you for that. I will try that soon.