Proxmox WebGUI not working

rakali

Active Member
Jan 2, 2020
42
5
28
42
My Proxmox webgui stopped working for some reason. It non-responsive.

I do get this error when looking at systemctl status pveproxy


Code:
Aug 10 07:29:57 menhir systemd[1]: Starting PVE API Proxy Server...
Aug 10 07:29:57 menhir pvecm[1942818]: unable to open file '/etc/pve/priv/authorized_keys.tmp.1942819' - Input/output error
Aug 10 07:29:58 menhir pveproxy[1942831]: starting server
Aug 10 07:29:58 menhir pveproxy[1942831]: starting 3 worker(s)
Aug 10 07:29:58 menhir pveproxy[1942831]: worker 1942832 started
Aug 10 07:29:58 menhir pveproxy[1942831]: worker 1942833 started
Aug 10 07:29:58 menhir pveproxy[1942831]: worker 1942834 started
Aug 10 07:29:58 menhir systemd[1]: Started PVE API Proxy Server.

Code:
# df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   7.3G     0  7.3G   0% /dev
tmpfs                                  4.7G   11M  4.7G   1% /run
rpool/ROOT/pve-1                       648G  311G  337G  48% /
tmpfs                                   24G   22M   24G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
rpool                                  337G  128K  337G   1% /rpool
rpool/ROOT                             337G  128K  337G   1% /rpool/ROOT
rpool/data                             337G  128K  337G   1% /rpool/data
rpool/data/subvol-103-disk-0           100G   42G   59G  42% /rpool/data/subvol-103-disk-0
/dev/fuse                              128M   24K  128M   1% /etc/pve

Code:
# ls -l /etc/pve/priv/
total 3
drwx------ 2 root www-data    0 Mar 23 12:26 acme
-rw------- 1 root www-data 1675 Aug  8 20:51 authkey.key
-rw------- 1 root www-data 1169 Aug  8 10:44 authorized_keys
-rw------- 1 root www-data 1167 Aug  8 10:44 known_hosts
drwx------ 2 root www-data    0 Mar 23 12:26 lock
-rw------- 1 root www-data 3243 Mar 23 12:26 pve-root-ca.key
-rw------- 1 root www-data    3 Mar 23 11:44 pve-root-ca.srl

Code:
pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-3-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-6
pve-kernel-helper: 7.0-6
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-3-pve: 5.11.22-6
pve-kernel-5.11.22-2-pve: 5.11.22-4
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.2.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-5
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-10
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.8-1
proxmox-backup-file-restore: 2.0.8-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
 
Last edited:
what about pmxcfs? is this with a cluster or a standalone node?

pvecm status
systemctl status pve-cluster corosync
 
Hi, sorry, I should have said, it is a standalone system. No cluster.

Code:
# pvecm status
Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?



Code:
# systemctl status pve-cluster corosync
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enable>
     Active: active (running) since Tue 2021-08-10 07:42:36 CEST; 4h>
    Process: 1964014 ExecStart=/usr/bin/pmxcfs (code=exited, status=>
   Main PID: 1964016 (pmxcfs)
      Tasks: 7 (limit: 17735)
     Memory: 16.9M
        CPU: 13.305s
     CGroup: /system.slice/pve-cluster.service
             └─1964016 /usr/bin/pmxcfs

Aug 10 07:42:35 menhir systemd[1]: Starting The Proxmox VE cluster f>
Aug 10 07:42:36 menhir systemd[1]: Started The Proxmox VE cluster fi>

● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/lib/systemd/system/corosync.service; enabled; >
     Active: inactive (dead)
  Condition: start condition failed at Tue 2021-08-10 07:43:52 CEST;>
             └─ ConditionPathExists=/etc/corosync/corosync.conf was >
       Docs: man:corosync
             man:corosync.conf
             man:corosync_overview

Aug 07 16:46:41 menhir systemd[1]: Condition check resulted in Coros>
Aug 10 07:23:36 menhir systemd[1]: Condition check resulted in Coros>
Aug 10 07:42:36 menhir systemd[1]: Condition check resulted in Coros>
Aug 10 07:43:52 menhir systemd[1]: Condition check resulted in Coros>
lines 1-27/27 (END)
 
okay - could you post the output of journalctl -b -u pve-cluster and df -h
 
Code:
# journalctl -b -u pve-cluster
-- Journal begins at Tue 2021-07-20 09:37:46 CEST, ends at Tue 2021-08-10 19:22:20 CEST. --
Aug 07 16:46:40 menhir systemd[1]: Starting The Proxmox VE cluster filesystem...
Aug 07 16:46:41 menhir systemd[1]: Started The Proxmox VE cluster filesystem.
Aug 09 14:34:25 menhir pmxcfs[3814]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Aug 09 14:34:25 menhir pmxcfs[3814]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Aug 09 14:34:25 menhir pmxcfs[3814]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/109: -1

Aug 09 15:23:49 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:23:49 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:23:49 menhir pmxcfs[3814]: [status] notice: RRDC update er>

Aug 09 15:35:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:35:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:35:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:35:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:36:30 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:36:30 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:36:30 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:36:30 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:38:07 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:39:13 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:05 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:40:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:28 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:41:58 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:43:01 menhir pmxcfs[3814]: [status] notice: RRDC update er>
Aug 09 15:59:08 menhir pmxcfs[3814]: [database] crit: commit transac>
Aug 09 15:59:08 menhir pmxcfs[3814]: [database] crit: rollback trans>
Aug 10 07:42:16 menhir systemd[1]: Stopping The Proxmox VE cluster f>
Aug 10 07:42:16 menhir pmxcfs[3814]: [main] notice: teardown filesys>
Aug 10 07:42:17 menhir pmxcfs[3814]: [main] notice: exit proxmox con>
Aug 10 07:42:17 menhir systemd[1]: pve-cluster.service: Succeeded.
Aug 10 07:42:17 menhir systemd[1]: Stopped The Proxmox VE cluster fi>
Aug 10 07:42:17 menhir systemd[1]: pve-cluster.service: Consumed 3mi>
Aug 10 07:42:35 menhir systemd[1]: Starting The Proxmox VE cluster f>
Aug 10 07:42:36 menhir systemd[1]: Started The Proxmox VE cluster fi>
lines 903-937/937 (END)


Code:
# df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   7.3G     0  7.3G   0% /dev
tmpfs                                  4.7G   11M  4.7G   1% /run
rpool/ROOT/pve-1                       512G  310G  202G  61% /
tmpfs                                   24G   28M   24G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
rpool                                  202G  128K  202G   1% /rpool
rpool/ROOT                             202G  128K  202G   1% /rpool/ROOT
rpool/data                             202G  128K  202G   1% /rpool/data
rpool/data/subvol-103-disk-0           100G   96G  4.6G  96% /rpool/data/subvol-103-disk-0
tmpfs                                  4.7G   12K  4.7G   1% /run/user/0
/dev/fuse                              128M   24K  128M   1% /etc/pve
root@menhir:~#
 
the log is cut off at the right side - but these two look suspicious:

Code:
Aug 09 15:59:08 menhir pmxcfs[3814]: [database] crit: commit transac>
Aug 09 15:59:08 menhir pmxcfs[3814]: [database] crit: rollback trans>

did you maybe run out of space on / at that point? since you already restarted the service yesterday - does access to pmxcfs work now? e.g., touch /etc/pve/authkey.pub? if it does, restarting pveproxy and pvedaemon (systemctl restart pveproxy pvedaemon) should get the GUI running again
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!