Can't access the webUI suddenly.

Savicks

New Member
Apr 14, 2024
4
0
1
I'm having an issue getting into my webui.

I can only access with ssh.
I use totp (google auth) and it has worked flawless untill now. For some reason I cannot login anymore.

I've tried rm /etc/pve/priv/tfa.cfg However this only spat out an permission issue.

Code:
cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: cribNUC
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.10.115
  }
  node {
    name: cribserver
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.10.165
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: Cribserver
  config_version: 2
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Code:
pvecm status
Cluster information
-------------------
Name:             Cribserver
Config Version:   2
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Thu Jun  6 20:45:53 2024
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1.36
Quorate:          No

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      1
Quorum:           2 Activity blocked
Flags:

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.10.165 (local)

Code:
journalctl --since -10min -u pvedaemon.service
Jun 06 20:42:25 cribserver pvedaemon[1605]: Cluster not quorate - extending auth key lifetime!
Jun 06 20:42:25 cribserver pvedaemon[1605]: <root@pam> successful auth for user 'root@pam'
root@cribserver:~# journalctl --since -15min -u pvedaemon.service
Jun 06 20:36:11 cribserver systemd[1]: Starting pvedaemon.service - PVE API Daemon...
Jun 06 20:36:13 cribserver pvedaemon[1604]: starting server
Jun 06 20:36:13 cribserver pvedaemon[1604]: starting 3 worker(s)
Jun 06 20:36:13 cribserver pvedaemon[1604]: worker 1605 started
Jun 06 20:36:13 cribserver pvedaemon[1604]: worker 1606 started
Jun 06 20:36:13 cribserver pvedaemon[1604]: worker 1607 started
Jun 06 20:36:13 cribserver systemd[1]: Started pvedaemon.service - PVE API Daemon.
Jun 06 20:37:44 cribserver pvedaemon[1607]: Cluster not quorate - extending auth key lifetime!
Jun 06 20:37:44 cribserver pvedaemon[1607]: <root@pam> successful auth for user 'root@pam'
Jun 06 20:37:51 cribserver pvedaemon[1605]: Cluster not quorate - extending auth key lifetime!
Jun 06 20:37:51 cribserver pvedaemon[1605]: authentication failure; rhost=::ffff:192.168.0.205 user=root@pam msg=unable to open file '/etc/pve/priv/tfa.cfg.tmp.1605' - Permission denied
Jun 06 20:38:03 cribserver pvedaemon[1607]: authentication failure; rhost=::ffff:192.168.0.205 user=root@pve msg=no such user ('root@pve')
Jun 06 20:38:45 cribserver pvedaemon[1606]: authentication failure; rhost=::ffff:192.168.0.205 user=root@pve msg=no such user ('root@pve')
Jun 06 20:38:53 cribserver pvedaemon[1606]: Cluster not quorate - extending auth key lifetime!
Jun 06 20:38:53 cribserver pvedaemon[1606]: <root@pam> successful auth for user 'root@pam'
Jun 06 20:38:59 cribserver pvedaemon[1606]: Cluster not quorate - extending auth key lifetime!
Jun 06 20:38:59 cribserver pvedaemon[1606]: authentication failure; rhost=::ffff:192.168.0.205 user=root@pam msg=unable to open file '/etc/pve/priv/tfa.cfg.tmp.1606' - Permission denied
Jun 06 20:39:03 cribserver pvedaemon[1605]: Cluster not quorate - extending auth key lifetime!
Jun 06 20:39:03 cribserver pvedaemon[1605]: <root@pam> successful auth for user 'root@pam'
Jun 06 20:39:12 cribserver pvedaemon[1605]: Cluster not quorate - extending auth key lifetime!
Jun 06 20:39:12 cribserver pvedaemon[1605]: authentication failure; rhost=::ffff:192.168.0.205 user=root@pam msg=unable to open file '/etc/pve/priv/tfa.cfg.tmp.1605' - Permission denied
Jun 06 20:42:25 cribserver pvedaemon[1605]: Cluster not quorate - extending auth key lifetime!
Jun 06 20:42:25 cribserver pvedaemon[1605]: <root@pam> successful auth for user 'root@pam'

Code:
systemctl | grep pve
  etc-pve.mount                                                                                                         loaded active     mounted         /etc/pve
  pve-cluster.service                                                                                                   loaded active     running         The Proxmox VE cluster filesystem
  pve-firewall.service                                                                                                  loaded active     running         Proxmox VE firewall
  pve-guests.service                                                                                                    loaded activating start     start PVE guests
  pve-ha-crm.service                                                                                                    loaded active     running         PVE Cluster HA Resource Manager Daemon
  pve-ha-lrm.service                                                                                                    loaded active     running         PVE Local HA Resource Manager Daemon
  pve-lxc-syscalld.service                                                                                              loaded active     running         Proxmox VE LXC Syscall Daemon
  pvebanner.service                                                                                                     loaded active     exited          Proxmox VE Login Banner
  pvedaemon.service                                                                                                     loaded active     running         PVE API Daemon
  pvefw-logger.service                                                                                                  loaded active     running         Proxmox VE firewall logger
  pvenetcommit.service                                                                                                  loaded active     exited          Commit Proxmox VE network changes
  pveproxy.service                                                                                                      loaded active     running         PVE API Proxy Server
  pvescheduler.service                                                                                                  loaded inactive   dead      start Proxmox VE scheduler
  pvestatd.service                                                                                                      loaded active     running         PVE Status Daemon
  dev-pve-swap.swap                                                                                                     loaded active     active          /dev/pve/swap
  pve-storage.target                                                                                                    loaded active     active          PVE Storage Target
  pve-daily-update.timer                                                                                                loaded active     waiting         Daily PVE download activities

I've tried rebooting multiple times and doing an update to no success.
Anyone able to identify the problem?
 
Hello @Savicks

The output you provided tells us:
a) You have a two-node cluster
b) Only one node is up and you don't have Quorum
c) (b) means that /etc/pve is R/O
d) You've attempted to login as root@pve - no such user exists
e) Login with root@pam (default user) is blocked by the absence of Quorum

Note that the last output "systemctl|grep pve" is not very helpful as it shows nothing about the actual state of those services.

Your path to recovery is one of the following:
a) recover second node to restore quorum
b) change number of required votes via pvecm

Good luck


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
  • Like
Reactions: Kingneutron
Also - when you get back up and running you should add a 3rd full node (to prevent split-brain) or a Qdevice. Possibly a raspberry pi on UPS power, or a VM running on a different pc (needs to be on 24/7 and have low network latency)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!