[SOLVED] Can't log in via GUI or connect via PDM

TrevorW · Aug 11, 2026

After my main cluster node died (3 node cluster) I've decided to go the Datacentre Manager Route. I had TFA enabled on the main node and have been unable to login via GUI to the remaining nodes. One of them I just bit the bullet and reinstalled and I've connected it to PDM, the other remaining alive host I can't log into via GUI, I've got an SSH terminal active, and separated it from the cluster by running

Code:

root@Proxb2:~# systemctl stop pve-cluster.service
root@Proxb2:~# /usr/bin/pmxcfs -l

Code:

Aug 11 17:35:47 Proxb2 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem>
Aug 11 17:35:47 Proxb2 pmxcfs[1747]: [main] notice: teardown filesystem
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [quorum] crit: quorum_finalize failed: CS_ERR_BAD_HANDLE
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_track_delete nodelist failed: CS_ERR_BAD_H>
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_track_delete version failed: CS_ERR_BAD_HA>
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [confdb] crit: cmap_finalize failed: CS_ERR_BAD_HANDLE
Aug 11 17:35:48 Proxb2 pmxcfs[1747]: [main] notice: exit proxmox configuration filesystem (0)
Aug 11 17:35:48 Proxb2 systemd[1]: pve-cluster.service: Deactivated successfully.
Aug 11 17:35:48 Proxb2 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Aug 11 17:35:48 Proxb2 systemd[1]: pve-cluster.service: Consumed 13min 42.246s CPU time, 89M memory>
lines 1-19/19 (END)

Now when I try to login to the GUI I'm no longer prompted for TFA I just get

Code:

Login failed:
authentication failure (401)
Please try again

Is this salvageable, or do I have to bite the bullet on this one too and reinstall? I've got backups of the VMs, but if I don't have to reinstall I'd rather not.

JohnnieKF · Aug 11, 2026

The 401 arriving before the TFA prompt means it's failing at the first stage, so this is a password/PAM or /etc/pve access problem, not your TFA token.

Could be that pvedaemon and pveproxy are still holding the old pmxcfs mount you removed so they can't read user.cfg / authkey.key. They don't pick up a manually started pmxcfs on their own.

Check here:
journalctl -u pvedaemon -u pveproxy --since "-15 min" | grep -iE 'auth|denied|permission|no such file'

TrevorW · Aug 11, 2026

Thanks Johnnie - that's saying that the pve cluster filesystem isn't online - isn't that what pve-cluster.service starts?
Although I did notice that I no longer have /etc/pve/priv/ that I can see

Code:

root@Proxb2:~# journalctl -u pvedaemon -u pveproxy --since "-15 min" | grep -iE 'auth|denied|permission|no such file'
Aug 11 18:36:01 Proxb2 pvedaemon[2380949]: authentication failure; rhost=::ffff:10.1.0.11 user=root@pam msg=cfs-lock 'authkey' error: pve cluster filesystem not online.
Aug 11 18:36:21 Proxb2 pvedaemon[2380948]: authentication failure; rhost=::ffff:10.1.0.11 user=root@pam msg=cfs-lock 'authkey' error: pve cluster filesystem not online.

JohnnieKF · Aug 11, 2026

Yeah, pve-cluster.service is what starts pmxcfs and pmxcfs is what mounts /etc/pve. That's why the login is failing - your error is cfs-lock 'authkey', and /etc/pve/priv/authkey.key is the key pvedaemon signs your session ticket with [0]. With the mount down it can't read it, so auth fails before TFA is ever reached.

[0] https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html#_files

TrevorW · Aug 11, 2026

Thanks Johnnie

So if can reinstate the missing pieces of /etc/pve (less the cluster and TFA info) I should be in a workable state (I've got Proxsave backups from when he cluster was intact)

JohnnieKF · Aug 11, 2026

Hold off on restoring. Your configs are all still in there - they're just not being exposed because nothing is mounting them. Copying files into /etc/pve while it's unmounted also won't help, since they'd land in the underlying directory and get shadowed the moment pmxcfs mounts over the top.

We need to verify the current quorum status. If the node has lost quorum, it explains exactly why it isn't syncing and why the mount fails or drops to read-only.

Can you post the output of this from the node:

Bash:

pvecm status

TrevorW · Aug 12, 2026

Thanks Johhnie

Code:

root@Proxb2:~# pvecm status
Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?
root@Proxb2:~#

The cluster was a 3 node cluster, the "main" node that was the gateway crashed with hardware problems - it's still offline, the second node I just reinstalled because it wasn't as critical and it was easy to reinstall, that node is now connected to Proxmox Datacentre Manager. This last remaining node has important VMs but I've got backups and the data of these VMs are separate from the OS's - so if I had to reinstall I wouldn't lose anything, I'd just rather not reinstall if I can avoid it.

The nodes weren't HA, the only storage they shared was for ISOs and backups, but the cluster was created before PDM was a thing, I had thought about migrating to PDM previously but the cluster wasn't giving me issues, so it was a case of let sleeping dogs lie.

So in summary I had 3 nodes, 1 is now dead - will fix later, 1 is reinstalled and running connected to PDM, 1 is still part of the old cluster and running, I have SSH access but not GUI access, I'd just like to blow the cluster away and run with PDM.

TrevorW · Aug 14, 2026

Is it possible to regain the ability to login, or do I have to reinstall? If I need to reinstall I'll have to do it over the weekend.

JohnnieKF · Aug 14, 2026

The best thing to do now is finish removing it from the defunct cluster in that case [0].

Kill the manual pmxcfs:
killall pmxcfs

Stop corosync since it's out of the cluster:
systemctl stop corosync && systemctl disable corosync

Start up now it's a standalone node:
systemctl start pve-cluster

Check the mount is back:
ls /etc/pve/priv/

[0] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_separate_node_without_reinstall

TrevorW · Aug 14, 2026

Thanks Johnnie its complaining about pmxcfs after running the first 2 commands

Code:

root@Proxb2:~# systemctl start pve-cluster
Job for pve-cluster.service failed because the control process exited with error code.
See "systemctl status pve-cluster.service" and "journalctl -xeu pve-cluster.service" for details.

Code:

root@Proxb2:~# See "systemctl status pve-cluster.service" and "journalctl -xeu pve-cluster.service" for details.
-bash: See: command not found
root@Proxb2:~# systemctl status pve-cluster.service
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/usr/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: activating (start) since Fri 2026-08-14 16:37:02 AWST; 6s ago
        Job: 5360500
 Invocation: b1147c1674d64e11af7a84d7d561e2e3
  Cntrl PID: 937359 (pmxcfs)
      Tasks: 1 (limit: 70844)
     Memory: 1.3M (peak: 1.7M)
        CPU: 16ms
     CGroup: /system.slice/pve-cluster.service
             └─937359 /usr/bin/pmxcfs

Aug 14 16:37:02 Proxb2 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is a>
Aug 14 16:37:02 Proxb2 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem>
Aug 14 16:37:02 Proxb2 pmxcfs[937359]: [main] notice: resolved node name 'Proxb2' to '10.1.0.142' f>
Aug 14 16:37:02 Proxb2 pmxcfs[937359]: [main] notice: resolved node name 'Proxb2' to '10.1.0.142' f>
Aug 14 16:37:02 Proxb2 pmxcfs[937359]: [main] notice: unable to acquire pmxcfs lock - trying again
Aug 14 16:37:02 Proxb2 pmxcfs[937359]: [main] notice: unable to acquire pmxcfs lock - trying again
l

Code:

root@Proxb2:~# journalctl -xeu pve-cluster.service
░░ The process' exit code is 'exited' and its exit status is 255.
Aug 14 16:37:53 Proxb2 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit pve-cluster.service has entered the 'failed' state with result 'exit-code'.
Aug 14 16:37:53 Proxb2 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster fil>
░░ Subject: A start job for unit pve-cluster.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pve-cluster.service has finished with a failure.
░░
░░ The job identifier is 5361468 and the job result is failed.
Aug 14 16:37:53 Proxb2 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is a>
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ Automatic restarting of the unit pve-cluster.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Aug 14 16:37:53 Proxb2 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem>
░░ Subject: A start job for unit pve-cluster.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pve-cluster.service has begun execution.
░░
░░ The job identifier is 5361710.
Aug 14 16:37:53 Proxb2 pmxcfs[937510]: [main] notice: resolved node name 'Proxb2' to '10.1.0.142' f>
Aug 14 16:37:53 Proxb2 pmxcfs[937510]: [main] notice: resolved node name 'Proxb2' to '10.1.0.142' f>
Aug 14 16:37:53 Proxb2 pmxcfs[937510]: [main] notice: unable to acquire pmxcfs lock - trying again
Aug 14 16:37:53 Proxb2 pmxcfs[937510]: [main] notice: unable to acquire pmxcfs lock - trying again
lines 3424-3457/3457 (END)

Code:

root@Proxb2:~# ls /etc/pve/priv/
ls: cannot access '/etc/pve/priv/': No such file or directory

Trying to manually restart pmxcfs also errors now

Code:

root@Proxb2:~# /usr/bin/pmxcfs -l
[main] notice: resolved node name 'Proxb2' to '10.1.0.142' for default node IP address
[main] notice: unable to acquire pmxcfs lock - trying again
[main] crit: unable to acquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)

JohnnieKF · Aug 14, 2026

Try stopping the cluster service entirely and then check if theres a process holding the lock
systemctl stop pve-cluster
pgrep -a pmxcfs

TrevorW · Aug 14, 2026

Thanks Johnnie

Code:

root@Proxb2:~# systemctl  stop pve-cluster
root@Proxb2:~# pgrep -a pmxcfs
3951383 /usr/bin/pmxcfs -l
root@Proxb2:~# kill 3951383
root@Proxb2:~# pgrep -a pmxcfs
root@Proxb2:~#

Code:

root@Proxb2:~# systemctl start pve-cluster.service
root@Proxb2:~# systemctl status pve-cluster.service
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/usr/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: active (running) since Fri 2026-08-14 16:58:05 AWST; 12s ago
 Invocation: a8c8d0caaf654cadbedeed5bf91160bb
    Process: 940702 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
   Main PID: 940706 (pmxcfs)
      Tasks: 5 (limit: 70844)
     Memory: 15M (peak: 15M)
        CPU: 47ms
     CGroup: /system.slice/pve-cluster.service
             └─940706 /usr/bin/pmxcfs

Aug 14 16:58:04 Proxb2 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem>
Aug 14 16:58:04 Proxb2 pmxcfs[940702]: [main] notice: resolved node name 'Proxb2' to '10.1.0.142' f>
Aug 14 16:58:04 Proxb2 pmxcfs[940702]: [main] notice: resolved node name 'Proxb2' to '10.1.0.142' f>
Aug 14 16:58:05 Proxb2 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
lines 1-16/16 (END)

Code:

root@Proxb2:~# ls -al /etc/pve/
total 12
drwxr-xr-x   2 root www-data     0 Jan  1  1970 .
drwxr-xr-x 102 root root     12288 Aug 14 01:25 ..
-r--r-----   1 root www-data   155 Jan  1  1970 .clusterlog
-rw-r-----   1 root www-data     2 Jan  1  1970 .debug
lrwxr-xr-x   1 root www-data    12 Jan  1  1970 local -> nodes/Proxb2
lrwxr-xr-x   1 root www-data    16 Jan  1  1970 lxc -> nodes/Proxb2/lxc
-r--r-----   1 root www-data    39 Jan  1  1970 .members
lrwxr-xr-x   1 root www-data    19 Jan  1  1970 openvz -> nodes/Proxb2/openvz
lrwxr-xr-x   1 root www-data    24 Jan  1  1970 qemu-server -> nodes/Proxb2/qemu-server
-r--r-----   1 root www-data   294 Jan  1  1970 .rrd
-r--r-----   1 root www-data  1120 Jan  1  1970 .version
drwxr-xr-x   2 root www-data     0 Aug 12 00:46 virtual-guest
-r--r-----   1 root www-data    18 Jan  1  1970 .vmlist
root@Proxb2:~#

No /etc/pve/priv but I do have a copy from a proxsave backup - I can try restoring that (minus the tfa.config) and see if that gets me gui access again.

JohnnieKF · Aug 14, 2026

That looks like all your configs have been reset given the size of the files.

Can you post the output of /var/lib/pve-cluster/

Failing that, since it's left the cluster you could restore your backup, I'm not familiar with that tool so you would need to check if it backs up the entire contents of /etc/pve/

TrevorW · Aug 14, 2026

Thanks Johnnie

Code:

root@Proxb2:~# ls -al /var/lib/pve-cluster/
total 156
drwxr-xr-x  2 root root   4096 Aug 11 21:23 .
drwxr-xr-x 41 root root   4096 May 23 08:05 ..
-rw-------  1 root root   4096 Aug 11 17:36 config.db
-rw-------  1 root root  32768 Aug 14 17:02 config.db-shm
-rw-------  1 root root 107152 Aug 14 17:02 config.db-wal
-rw-------  1 root root      0 May 13  2023 .pmxcfs.lockfile
root@Proxb2:~#

Yeah, it backs up the entire contents of /etc/pve (https://github.com/tis24dev/proxsave)

JohnnieKF · Aug 14, 2026

Yeah looks like the db is in the same position, go ahead and restore with proxsave following their steps

TrevorW · Aug 14, 2026

Thanks Johnnie

TrevorW · Aug 15, 2026

After restoring from Proxsave I was able to follow the instructions in https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_separate_node_without_reinstall and recover the node from the failed cluster and connect it to PDM

[SOLVED] Can't log in via GUI or connect via PDM

TrevorW

Member

JohnnieKF

Member

TrevorW

Member

JohnnieKF

Member

TrevorW

Member

JohnnieKF

Member

TrevorW

Member

TrevorW

Member

JohnnieKF

Member

TrevorW

Member

JohnnieKF

Member

TrevorW

Member

JohnnieKF

Member

TrevorW

Member

JohnnieKF

Member

TrevorW

Member

TrevorW

Member

We value your privacy