Proxmox WebGUI not working

rakali

Active Member
Jan 2, 2020
42
5
28
43
My Proxmox webgui stopped working for some reason. It non-responsive.

I do get this error when looking at systemctl status pveproxy


Code:
Aug 10 07:29:57 menhir systemd[1]: Starting PVE API Proxy Server...
Aug 10 07:29:57 menhir pvecm[1942818]: unable to open file '/etc/pve/priv/authorized_keys.tmp.1942819' - Input/output error
Aug 10 07:29:58 menhir pveproxy[1942831]: starting server
Aug 10 07:29:58 menhir pveproxy[1942831]: starting 3 worker(s)
Aug 10 07:29:58 menhir pveproxy[1942831]: worker 1942832 started
Aug 10 07:29:58 menhir pveproxy[1942831]: worker 1942833 started
Aug 10 07:29:58 menhir pveproxy[1942831]: worker 1942834 started
Aug 10 07:29:58 menhir systemd[1]: Started PVE API Proxy Server.

Code:
# df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   7.3G     0  7.3G   0% /dev
tmpfs                                  4.7G   11M  4.7G   1% /run
rpool/ROOT/pve-1                       648G  311G  337G  48% /
tmpfs                                   24G   22M   24G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
rpool                                  337G  128K  337G   1% /rpool
rpool/ROOT                             337G  128K  337G   1% /rpool/ROOT
rpool/data                             337G  128K  337G   1% /rpool/data
rpool/data/subvol-103-disk-0           100G   42G   59G  42% /rpool/data/subvol-103-disk-0
/dev/fuse                              128M   24K  128M   1% /etc/pve

Code:
# ls -l /etc/pve/priv/
total 3
drwx------ 2 root www-data    0 Mar 23 12:26 acme
-rw------- 1 root www-data 1675 Aug  8 20:51 authkey.key
-rw------- 1 root www-data 1169 Aug  8 10:44 authorized_keys
-rw------- 1 root www-data 1167 Aug  8 10:44 known_hosts
drwx------ 2 root www-data    0 Mar 23 12:26 lock
-rw------- 1 root www-data 3243 Mar 23 12:26 pve-root-ca.key
-rw------- 1 root www-data    3 Mar 23 11:44 pve-root-ca.srl

Code:
pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-3-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-6
pve-kernel-helper: 7.0-6
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-3-pve: 5.11.22-6
pve-kernel-5.11.22-2-pve: 5.11.22-4
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.2.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-5
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-10
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.8-1
proxmox-backup-file-restore: 2.0.8-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
 
Last edited:
what about pmxcfs? is this with a cluster or a standalone node?

pvecm status
systemctl status pve-cluster corosync
 
Hi, sorry, I should have said, it is a standalone system. No cluster.

Code:
# pvecm status
Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?



Code:
# systemctl status pve-cluster corosync
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enable>
     Active: active (running) since Tue 2021-08-10 07:42:36 CEST; 4h>
    Process: 1964014 ExecStart=/usr/bin/pmxcfs (code=exited, status=>
   Main PID: 1964016 (pmxcfs)
      Tasks: 7 (limit: 17735)
     Memory: 16.9M
        CPU: 13.305s
     CGroup: /system.slice/pve-cluster.service
             └─1964016 /usr/bin/pmxcfs

Aug 10 07:42:35 menhir systemd[1]: Starting The Proxmox VE cluster f>
Aug 10 07:42:36 menhir systemd[1]: Started The Proxmox VE cluster fi>

● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/lib/systemd/system/corosync.service; enabled; >
     Active: inactive (dead)
  Condition: start condition failed at Tue 2021-08-10 07:43:52 CEST;>
             └─ ConditionPathExists=/etc/corosync/corosync.conf was >
       Docs: man:corosync
             man:corosync.conf
             man:corosync_overview

Aug 07 16:46:41 menhir systemd[1]: Condition check resulted in Coros>
Aug 10 07:23:36 menhir systemd[1]: Condition check resulted in Coros>
Aug 10 07:42:36 menhir systemd[1]: Condition check resulted in Coros>
Aug 10 07:43:52 menhir systemd[1]: Condition check resulted in Coros>
lines 1-27/27 (END)
 
okay - could you post the output of journalctl -b -u pve-cluster and df -h
 
Code:
# journalctl -b -u pve-cluster
-- Journal begins at Tue 2021-07-20 09:37:46 CEST, ends at Tue 2021-08-10 19:22:20 CEST. --
Aug 07 16:46:40 menhir systemd[1]: Starting The Proxmox VE cluster filesystem...
Aug 07 16:46:41 menhir systemd[1]: Started The Proxmox VE cluster filesystem.
Aug 09 14:34:25 menhir pmxcfs[3814]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Aug 09 14:34:25 menhir pmxcfs[3814]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Aug 09 14:34:25 menhir pmxcfs[3814]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/109: -1

Aug 09 15:23:49 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:23:49 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:23:49 menhir pmxcfs[3814]: [status] notice: RRDC update er>

Aug 09 15:35:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:35:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:35:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:35:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:36:30 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:36:30 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:36:30 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:36:30 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:59:08 menhir pmxcfs[3814]: [database] crit: commit transac>
Aug 09 15:59:08 menhir pmxcfs[3814]: [database] crit: rollback trans>
Aug 10 07:42:16 menhir systemd[1]: Stopping The Proxmox VE cluster f>
Aug 10 07:42:16 menhir pmxcfs[3814]: [main] notice: teardown filesys>
Aug 10 07:42:17 menhir pmxcfs[3814]: [main] notice: exit proxmox con>
Aug 10 07:42:17 menhir systemd[1]: pve-cluster.service: Succeeded.
Aug 10 07:42:17 menhir systemd[1]: Stopped The Proxmox VE cluster fi>
Aug 10 07:42:17 menhir systemd[1]: pve-cluster.service: Consumed 3mi>
Aug 10 07:42:35 menhir systemd[1]: Starting The Proxmox VE cluster f>
Aug 10 07:42:36 menhir systemd[1]: Started The Proxmox VE cluster fi>
lines 903-937/937 (END)


Code:
# df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   7.3G     0  7.3G   0% /dev
tmpfs                                  4.7G   11M  4.7G   1% /run
rpool/ROOT/pve-1                       512G  310G  202G  61% /
tmpfs                                   24G   28M   24G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
rpool                                  202G  128K  202G   1% /rpool
rpool/ROOT                             202G  128K  202G   1% /rpool/ROOT
rpool/data                             202G  128K  202G   1% /rpool/data
rpool/data/subvol-103-disk-0           100G   96G  4.6G  96% /rpool/data/subvol-103-disk-0
tmpfs                                  4.7G   12K  4.7G   1% /run/user/0
/dev/fuse                              128M   24K  128M   1% /etc/pve
root@menhir:~#
 
the log is cut off at the right side - but these two look suspicious:

Code:
Aug 09 15:59:08 menhir pmxcfs[3814]: [database] crit: commit transac>
Aug 09 15:59:08 menhir pmxcfs[3814]: [database] crit: rollback trans>

did you maybe run out of space on / at that point? since you already restarted the service yesterday - does access to pmxcfs work now? e.g., touch /etc/pve/authkey.pub? if it does, restarting pveproxy and pvedaemon (systemctl restart pveproxy pvedaemon) should get the GUI running again