Webinterface gone after blackout?

t3ngu · Jul 20, 2017

Hey guys,

a bit of an unusual problem: My Proxmox-server lost power last night after a voltage peak raged in our serverroom. Unfortunately the Proxmox-Server wasn't behind a battery-backup, so it was turned off hard.
The damage is manageable, in fact, the only thing that happend to the Proxmox-setup was the loss of 2 harddrives... and the webinterface.
The Server is running right now, all machines that are flagged as autostart are up and running, I can ssh into the Proxmox-host and all servers and appaerently no data loss occured. But, the webfrontend won't respond. If I try to reach it in my browser via https://192.168.2.101:8006/ (yep thats the right IP), Firefox states in the lower left corner of the Window "Starting TLS-Handshake with 192.168.2.101..." and thats pretty much it. No error or anything, not even a timeout. I don't have a SSL-certificate for Proxmox, but that was never a problem before. I cleared all my cookies and tried different brwosers, no luck.

I did soome searching and now I am sceptical that a webserer is still installed and runnig on my Proxmox. Wasn't Proxmox using Apache for the webfrontend? "dpkg -l | grep apache" shows nothing on my machine. Is there any way to verify my webserver is there, up and running? Or has anybody other thougts on this?
Thanks in advance!

dcsapak · Jul 21, 2017

what version do you run? we do not use apache anymore for a long time

what is the output of

Code:

systemctl status pveproxy pvedaemon

t3ngu · Jul 21, 2017

Hey, thanks for the answer and good to know that Apache is not the problem.
pveproxy and pvedaemon are up and running, here is the output:

Code:

● pveproxy.service - PVE API Proxy Server
   Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled)
   Active: active (running) since Thu 2017-07-20 22:50:43 CEST; 13h ago
  Process: 13871 ExecStop=/usr/bin/pveproxy stop (code=exited, status=0/SUCCESS)
  Process: 12273 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
  Process: 13892 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
 Main PID: 13896 (pveproxy)
   CGroup: /system.slice/pveproxy.service
           ├─  877 pveproxy worker
           ├─  880 pveproxy worker
           ├─  884 pveproxy worker
           └─13896 pveproxy

● pvedaemon.service - PVE API Daemon
   Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled)
   Active: active (running) since Thu 2017-07-20 15:56:47 CEST; 20h ago
 Main PID: 1250 (pvedaemon)
   CGroup: /system.slice/pvedaemon.service
           ├─1250 pvedaemon
           ├─1251 pvedaemon worker
           ├─1252 pvedaemon worker
           └─1253 pvedaemon worker

Edit: Another thing: Is qm still supposed to be a way to interact via commandline with installd VMs? I tried "qm list", but proxmox repsonded:

Code:

-bash: qm: command not found

. Should I be worried or is this another thing that has been outsorted?
Edit 2: Solved, I wasn't logged in as root and therefore not in the qm-group.

t3ngu · Jul 22, 2017

Something new: When i stop pveproxy on my proxmox via systemctl and try to reach the webgui, my browser immediately state that this website is not reachable. If i turn pveproxy back on again, firefox stucks in its endless loop, but chromium gives a timeout after some time.

So, pveproxy is there, and kind of responsive, because it makes a difference if it's turned on or off. But somehow, even when turned on, it fails to accept connections. Is there any way to reinstall/reconfigure pveproxy or put it back to default settings? Any other ideas on this?

udo · Jul 22, 2017

t3ngu said:
Something new: When i stop pveproxy on my proxmox via systemctl and try to reach the webgui, my browser immediately state that this website is not reachable. If i turn pveproxy back on again, firefox stucks in its endless loop, but chromium gives a timeout after some time.

So, pveproxy is there, and kind of responsive, because it makes a difference if it's turned on or off. But somehow, even when turned on, it fails to accept connections. Is there any way to reinstall/reconfigure pveproxy or put it back to default settings? Any other ideas on this?

Hi,
to help we need a little bit more input.

Can you do following commands and post the output?

Code:

pveversion -v
ss -patn | grep 8006

sytemctl restart pveproxy
tail -30 /var/log/syslog

# try to access the webgui https://192.168.2.101:8006
# because we need logentrys

tail -20 /var/log/pveproxy/access.log
tail /var/log/pve-firewall.log

dpkg -l | grep pve

Udo

t3ngu · Jul 23, 2017

Hey Udo, thanks for the reply!

Here's the output of the commands:

pveversion -v

Code:

proxmox-ve: 4.4-92 (running kernel: 4.4.67-1-pve)
pve-manager: 4.4-15 (running version: 4.4-15/7599e35a)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.59-1-pve: 4.4.59-87
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.62-1-pve: 4.4.62-88
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-52
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-95
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-100
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80

ss before/after I tried to connect to the webpage:

Code:

ss -patn | grep 8006
LISTEN     0      128          *:8006                     *:*

ss -patn | grep 8006
LISTEN     1      128          *:8006                     *:*                
ESTAB      168    0      192.168.2.101:8006               192.168.2.102:52114

Syslog after restarting pveproxy:

Code:

Jul 23 11:18:54 marduk pveproxy[13676]: starting 2 worker(s)
Jul 23 11:18:54 marduk pveproxy[13676]: worker 13691 started
Jul 23 11:18:54 marduk pveproxy[13690]: unable to open log file '/var/log/pveproxy/access.log' - No such file or directory
Jul 23 11:18:54 marduk pveproxy[13676]: worker 13692 started
Jul 23 11:18:54 marduk pveproxy[13691]: unable to open log file '/var/log/pveproxy/access.log' - No such file or directory
Jul 23 11:18:54 marduk pveproxy[13692]: unable to open log file '/var/log/pveproxy/access.log' - No such file or directory
Jul 23 11:18:54 marduk spiceproxy[13680]: worker exit
Jul 23 11:18:54 marduk spiceproxy[1276]: worker 13680 finished
Jul 23 11:18:54 marduk spiceproxy[1276]: starting 1 worker(s)
Jul 23 11:18:54 marduk spiceproxy[1276]: worker 13693 started
Jul 23 11:18:54 marduk spiceproxy[13693]: unable to open log file '/var/log/pveproxy/access.log' - No such file or directory
Jul 23 11:18:59 marduk pveproxy[13690]: worker exit
Jul 23 11:18:59 marduk pveproxy[13691]: worker exit
Jul 23 11:18:59 marduk pveproxy[13692]: worker exit
Jul 23 11:18:59 marduk pveproxy[13676]: worker 13690 finished
Jul 23 11:18:59 marduk pveproxy[13676]: starting 1 worker(s)
Jul 23 11:18:59 marduk pveproxy[13676]: worker 13691 finished
Jul 23 11:18:59 marduk pveproxy[13676]: worker 13694 started
Jul 23 11:18:59 marduk pveproxy[13676]: worker 13692 finished
Jul 23 11:18:59 marduk pveproxy[13676]: starting 2 worker(s)
Jul 23 11:18:59 marduk pveproxy[13694]: unable to open log file '/var/log/pveproxy/access.log' - No such file or directory
Jul 23 11:18:59 marduk pveproxy[13676]: worker 13695 started
Jul 23 11:18:59 marduk pveproxy[13676]: worker 13696 started
Jul 23 11:18:59 marduk pveproxy[13695]: unable to open log file '/var/log/pveproxy/access.log' - No such file or directory
Jul 23 11:18:59 marduk pveproxy[13696]: unable to open log file '/var/log/pveproxy/access.log' - No such file or directory
Jul 23 11:18:59 marduk spiceproxy[13693]: worker exit
Jul 23 11:18:59 marduk spiceproxy[1276]: worker 13693 finished
Jul 23 11:18:59 marduk spiceproxy[1276]: starting 1 worker(s)
Jul 23 11:18:59 marduk spiceproxy[1276]: worker 13697 started
Jul 23 11:18:59 marduk spiceproxy[13697]: unable to open log file '/var/log/pveproxy/access.log' - No such file or directory

So... the access.log seems to be gone, can I just touch a new one?

pve-firewall.log:

Code:

tail /var/log/pve-firewall.log
0 5 - 23/Jul/2017:06:25:37 +0200 starting pvefw logger

And last but not least te grep of 'pve':

Code:

dpkg -l | grep pve
ii  corosync-pve                   2.4.2-2~pve4+1                     amd64        Standards-based cluster framework (daemon and modules)
ii  dmeventd                       2:1.02.93-pve3                     amd64        Linux Kernel Device Mapper event daemon
ii  dmsetup                        2:1.02.93-pve3                     amd64        Linux Kernel Device Mapper userspace library
ii  grub-common                    2.02-pve5                          amd64        GRand Unified Bootloader (common files)
ii  grub-efi-amd64-bin             2.02-pve5                          amd64        GRand Unified Bootloader, version 2 (EFI-AMD64 binaries)
ii  grub-efi-ia32-bin              2.02-pve5                          amd64        GRand Unified Bootloader, version 2 (EFI-IA32 binaries)
ii  grub-pc                        2.02-pve5                          amd64        GRand Unified Bootloader, version 2 (PC/BIOS version)
ii  grub-pc-bin                    2.02-pve5                          amd64        GRand Unified Bootloader, version 2 (PC/BIOS binaries)
ii  grub2-common                   2.02-pve5                          amd64        GRand Unified Bootloader (common files for version 2)
ii  libcorosync4-pve               2.4.2-2~pve4+1                     amd64        Standards-based cluster framework (libraries)
ii  libdevmapper-event1.02.1:amd64 2:1.02.93-pve3                     amd64        Linux Kernel Device Mapper event support library
ii  libdevmapper1.02.1:amd64       2:1.02.93-pve3                     amd64        Linux Kernel Device Mapper userspace library
ii  liblvm2app2.2:amd64            2.02.116-pve3                      amd64        LVM2 application library
ii  liblvm2cmd2.02:amd64           2.02.116-pve3                      amd64        LVM2 command library
ii  libnvpair1linux                0.6.5.9-pve15~bpo80                amd64        Solaris name-value library for Linux
ii  libpve-access-control          4.0-23                             amd64        Proxmox VE access control library
ii  libpve-common-perl             4.0-95                             all          Proxmox VE base library
ii  libpve-guest-common-perl       1.0-2                              all          Proxmox VE common guest-related modules
ii  libpve-http-server-perl        1.0-4                              all          Proxmox Asynchrounous HTTP Server Implementation
ii  libpve-storage-perl            4.0-76                             all          Proxmox VE storage management library
ii  libuutil1linux                 0.6.5.9-pve15~bpo80                amd64        Solaris userland utility library for Linux
ii  libzfs2linux                   0.6.5.9-pve15~bpo80                amd64        OpenZFS filesystem library for Linux
ii  libzpool2linux                 0.6.5.9-pve15~bpo80                amd64        OpenZFS pool library for Linux
ii  lvm2                           2.02.116-pve3                      amd64        Linux Logical Volume Manager
ii  lxc-pve                        2.0.7-4                            amd64        Linux containers usersapce tools
ii  lxcfs                          2.0.6-pve1                         amd64        LXC userspace filesystem
ii  novnc-pve                      0.5-9                              amd64        HTML5 VNC client
ii  pve-cluster                    4.0-52                             amd64        Cluster Infrastructure for Proxmox Virtual Environment
ii  pve-container                  1.0-100                            all          Proxmox VE Container management tool
ii  pve-docs                       4.4-4                              all          Proxmox VE Documentation
ii  pve-firewall                   2.0-33                             amd64        Proxmox VE Firewall
ii  pve-firmware                   1.1-11                             all          Binary firmware code for the pve-kernel
ii  pve-ha-manager                 1.0-41                             amd64        Proxmox VE HA Manager
ii  pve-kernel-4.4.35-1-pve        4.4.35-77                          amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-4.4.49-1-pve        4.4.49-86                          amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-4.4.59-1-pve        4.4.59-87                          amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-4.4.62-1-pve        4.4.62-88                          amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-4.4.67-1-pve        4.4.67-92                          amd64        The Proxmox PVE Kernel Image
ii  pve-libspice-server1           0.12.8-2                           amd64        SPICE remote display system server library
ii  pve-manager                    4.4-15                             amd64        The Proxmox Virtual Environment
ii  pve-qemu-kvm                   2.7.1-4                            amd64        Full virtualization on x86 hardware
ii  smartmontools                  6.5+svn4324-1~pve80                amd64        control and monitor storage systems using S.M.A.R.T.
ii  spl                            0.6.5.9-pve8~bpo80                 amd64        Solaris Porting Layer user-space utilities for Linux
ii  tar                            1.27.1+pve.3                       amd64        GNU version of the tar archiving utility
ii  zfs-initramfs                  0.6.5.9-pve15~bpo80                all          OpenZFS root filesystem capabilities for Linux - initramfs
ii  zfsutils                       0.6.5.9-pve15~bpo80                all          transitional package
ii  zfsutils-linux                 0.6.5.9-pve15~bpo80                amd64        command-line tools to manage OpenZFS filesystems

Thanks for your help!

udo · Jul 23, 2017

Hi,
is the permission of the pveproxy-dir correct?

Code:

ls -lsa /var/log/pveproxy/

The correct permissions looks like this:

Code:

ls -lsa /var/log/pveproxy/
total 44
 4 drwx------  2 www-data www-data 4096 Jul 23 06:25 .
 4 drwxr-xr-x 13 root     root     4096 Jul 23 06:25 ..
 0 -rw-r-----  1 www-data www-data    0 Jul 23 06:25 access.log
...

Udo

t3ngu · Jul 23, 2017

Hey Udo,
well - thats a problem, because /var/log/pveproxy/ doesnt seem to exist on my server:

Code:

ls -lsa /var/log/pveproxy/
ls: cannot access /var/log/pveproxy/: No such file or directory

What do I do now?

udo · Jul 23, 2017

Hi,
are there others pve-dirs also gone?

Code:

find /var/log -type d

If there an /var/log/pve dir, simply create thoe pveproxy-dir:

Code:

mkdir /var/log/pveproxy
chmod 700 /var/log/pveproxy
chown www-data:www-data /var/log/pveproxy

systemctl restart pveproxy

Udo

t3ngu · Jul 23, 2017

Thanks very much - that was it! I created /var/log/pveproxy again and now the webgui is accessibel again!

Search

Search

Webinterface gone after blackout?

t3ngu

New Member

dcsapak

Proxmox Staff Member

t3ngu

New Member

t3ngu

New Member

udo

Distinguished Member

t3ngu

New Member

udo

Distinguished Member

t3ngu

New Member

udo

Distinguished Member

t3ngu

New Member