Can't log in via GUI or connect via PDM

[TrevorW]

After my main cluster node died (3 node cluster) I've decided to go the Datacentre Manager Route. I had TFA enabled on the main node and have been unable to login via GUI to the remaining nodes. One of them I just bit the bullet and reinstalled and I've connected it to PDM, the other remaining alive host I can't log into via GUI, I've got an SSH terminal active, and separated it from the cluster by running

root@Proxb2:~# systemctl stop pve-cluster.service
root@Proxb2:~# /usr/bin/pmxcfs -l

Aug 11 17:35:47 Proxb2 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem>
Aug 11 17:35:47 Proxb2 pmxcfs[1747]: [main] notice: teardown filesystem
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [quorum] crit: quorum_finalize failed: CS_ERR_BAD_HANDLE
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_track_delete nodelist failed: CS_ERR_BAD_H>
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_track_delete version failed: CS_ERR_BAD_HA>
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_finalize failed: CS_ERR_BAD_HANDLE
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [main] notice: exit proxmox configuration filesystem (0)
Aug 11 17:35:48 Proxb2 systemd[1]: pve-cluster.service: Deactivated successfully.
Aug 11 17:35:48 Proxb2 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Aug 11 17:35:48 Proxb2 systemd[1]: pve-cluster.service: Consumed 13min 42.246s CPU time, 89M memory>
lines 1-19/19 (END)

Now when I try to login to the GUI I'm no longer prompted for TFA I just get

Login failed:
authentication failure (401)
Please try again

Is this salvageable, or do I have to bite the bullet on this one too and reinstall? I've got backups of the VMs, but if I don't have to reinstall I'd rather not.

 

[JohnnieKF]

The 401 arriving before the TFA prompt means it's failing at the first stage, so this is a password/PAM or /etc/pve access problem, not your TFA token.

Could be that pvedaemon and pveproxy are still holding the old pmxcfs mount you removed so they can't read user.cfg / authkey.key. They don't pick up a manually started pmxcfs on their own.

Check here:
journalctl -u pvedaemon -u pveproxy --since "-15 min" | grep -iE 'auth|denied|permission|no such file'

 

[TrevorW]

Thanks Johnnie - that's saying that the pve cluster filesystem isn't online - isn't that what pve-cluster.service starts?
Although I did notice that I no longer have /etc/pve/priv/ that I can see

root@Proxb2:~# journalctl -u pvedaemon -u pveproxy --since "-15 min" | grep -iE 'auth|denied|permission|no such file'
Aug 11 18:36:01 Proxb2 pvedaemon[2380949]: authentication failure; rhost=::ffff:10.1.0.11 user=root@pam msg=cfs-lock 'authkey' error: pve cluster filesystem not online.
Aug 11 18:36:21 Proxb2 pvedaemon[2380948]: authentication failure; rhost=::ffff:10.1.0.11 user=root@pam msg=cfs-lock 'authkey' error: pve cluster filesystem not online.

 

[JohnnieKF]

Yeah, pve-cluster.service is what starts pmxcfs and pmxcfs is what mounts /etc/pve. That's why the login is failing - your error is cfs-lock 'authkey', and /etc/pve/priv/authkey.key is the key pvedaemon signs your session ticket with [0]. With the mount down it can't read it, so auth fails before TFA is ever reached.

[0] https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html#_files

 

[TrevorW]

Thanks Johnnie

So if can reinstate the missing pieces of /etc/pve (less the cluster and TFA info) I should be in a workable state (I've got Proxsave backups from when he cluster was intact)

 

[JohnnieKF]

Hold off on restoring. Your configs are all still in there - they're just not being exposed because nothing is mounting them. Copying files into /etc/pve while it's unmounted also won't help, since they'd land in the underlying directory and get shadowed the moment pmxcfs mounts over the top.

We need to verify the current quorum status. If the node has lost quorum, it explains exactly why it isn't syncing and why the mount fails or drops to read-only.

Can you post the output of this from the node:

pvecm status

 

[TrevorW]

Thanks Johhnie

root@Proxb2:~# pvecm status
Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?
root@Proxb2:~#

The cluster was a 3 node cluster, the "main" node that was the gateway crashed with hardware problems - it's still offline, the second node I just reinstalled because it wasn't as critical and it was easy to reinstall, that node is now connected to Proxmox Datacentre Manager. This last remaining node has important VMs but I've got backups and the data of these VMs are separate from the OS's - so if I had to reinstall I wouldn't lose anything, I'd just rather not reinstall if I can avoid it.

The nodes weren't HA, the only storage they shared was for ISOs and backups, but the cluster was created before PDM was a thing, I had thought about migrating to PDM previously but the cluster wasn't giving me issues, so it was a case of let sleeping dogs lie.

So in summary I had 3 nodes, 1 is now dead - will fix later, 1 is reinstalled and running connected to PDM, 1 is still part of the old cluster and running, I have SSH access but not GUI access, I'd just like to blow the cluster away and run with PDM.