Can't log in via GUI or connect via PDM

[TrevorW]

After my main cluster node died (3 node cluster) I've decided to go the Datacentre Manager Route. I had TFA enabled on the main node and have been unable to login via GUI to the remaining nodes. One of them I just bit the bullet and reinstalled and I've connected it to PDM, the other remaining alive host I can't log into via GUI, I've got an SSH terminal active, and separated it from the cluster by running

root@Proxb2:~# systemctl stop pve-cluster.service
root@Proxb2:~# /usr/bin/pmxcfs -l

Aug 11 17:35:47 Proxb2 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem>
Aug 11 17:35:47 Proxb2 pmxcfs[1747]: [main] notice: teardown filesystem
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [quorum] crit: quorum_finalize failed: CS_ERR_BAD_HANDLE
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_track_delete nodelist failed: CS_ERR_BAD_H>
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_track_delete version failed: CS_ERR_BAD_HA>
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_finalize failed: CS_ERR_BAD_HANDLE
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [main] notice: exit proxmox configuration filesystem (0)
Aug 11 17:35:48 Proxb2 systemd[1]: pve-cluster.service: Deactivated successfully.
Aug 11 17:35:48 Proxb2 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Aug 11 17:35:48 Proxb2 systemd[1]: pve-cluster.service: Consumed 13min 42.246s CPU time, 89M memory>
lines 1-19/19 (END)

Now when I try to login to the GUI I'm no longer prompted for TFA I just get

Login failed:
authentication failure (401)
Please try again

Is this salvageable, or do I have to bite the bullet on this one too and reinstall? I've got backups of the VMs, but if I don't have to reinstall I'd rather not.

 

[JohnnieKF]

The 401 arriving before the TFA prompt means it's failing at the first stage, so this is a password/PAM or /etc/pve access problem, not your TFA token.

Could be that pvedaemon and pveproxy are still holding the old pmxcfs mount you removed so they can't read user.cfg / authkey.key. They don't pick up a manually started pmxcfs on their own.

Check here:
journalctl -u pvedaemon -u pveproxy --since "-15 min" | grep -iE 'auth|denied|permission|no such file'

 

[TrevorW]

Thanks Johnnie - that's saying that the pve cluster filesystem isn't online - isn't that what pve-cluster.service starts?
Although I did notice that I no longer have /etc/pve/priv/ that I can see

root@Proxb2:~# journalctl -u pvedaemon -u pveproxy --since "-15 min" | grep -iE 'auth|denied|permission|no such file'
Aug 11 18:36:01 Proxb2 pvedaemon[2380949]: authentication failure; rhost=::ffff:10.1.0.11 user=root@pam msg=cfs-lock 'authkey' error: pve cluster filesystem not online.
Aug 11 18:36:21 Proxb2 pvedaemon[2380948]: authentication failure; rhost=::ffff:10.1.0.11 user=root@pam msg=cfs-lock 'authkey' error: pve cluster filesystem not online.

 

[JohnnieKF]

Yeah, pve-cluster.service is what starts pmxcfs and pmxcfs is what mounts /etc/pve. That's why the login is failing - your error is cfs-lock 'authkey', and /etc/pve/priv/authkey.key is the key pvedaemon signs your session ticket with [0]. With the mount down it can't read it, so auth fails before TFA is ever reached.

[0] https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html#_files

 

[TrevorW]

Thanks Johnnie

So if can reinstate the missing pieces of /etc/pve (less the cluster and TFA info) I should be in a workable state (I've got Proxsave backups from when he cluster was intact)

 

[JohnnieKF]

Hold off on restoring. Your configs are all still in there - they're just not being exposed because nothing is mounting them. Copying files into /etc/pve while it's unmounted also won't help, since they'd land in the underlying directory and get shadowed the moment pmxcfs mounts over the top.

We need to verify the current quorum status. If the node has lost quorum, it explains exactly why it isn't syncing and why the mount fails or drops to read-only.

Can you post the output of this from the node:

pvecm status