[SOLVED] Authentication failure - invalid PVE ticket

[JohanKarlsson]

Hi,

Got a very strange issue with one of my PVE clusters currently. Logging in through the web UI yields the error message for users with 2FA:

Login failed. Please try again

In journalctl the pvedaemon service also logs the following error:

authentication failure; rhost=::ffff:123.456.789.123 user=root@pam msg=401 permission denied - invalid PVE ticket

For users without 2FA the sessions instead seem to expire instantly.
The issue can be fixed for a while by running touch /etc/pve/authkey.pub and touch /etc/pve/authkey.pub.old but always reappears shortly.

Full quorum is established and all system clocks synced via NTP. Time and date has been verified matching with my local time and `hwclock`
This is the type of behaviour that I would except if the clocks were out of sync, but they're not...

Cluster has full quorum and shows healthy, all pve related services have been restarted as well without success.

Thanks in advance,
Johan

 

[JohanKarlsson]

Adding on the output of pveversion -v to this:

proxmox-ve: 7.4-1 (running kernel: 5.15.152-1-pve)
pve-manager: 7.4-18 (running version: 7.4-18/b1f94095)
pve-kernel-5.15: 7.4-13
pve-kernel-5.15.152-1-pve: 5.15.152-1
pve-kernel-5.15.149-1-pve: 5.15.149-1
pve-kernel-5.15.126-1-pve: 5.15.126-1
pve-kernel-5.15.53-1-pve: 5.15.53-1
ceph-fuse: 17.2.1-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx4
ksmtuned: 4.20150326
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.3
libpve-apiclient-perl: 3.2-2
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-network-perl: 0.7.3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.6-1
proxmox-backup-file-restore: 2.4.6-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.4
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+3
pve-firewall: 4.3-5
pve-firmware: 3.6-6
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.10-1
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-6
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.15-pve1

 

[JohanKarlsson]

Resolved!

Had to, in a specific order combined with luck (?) do the following:

1. Stop corosync and pve-cluster
2. Stop pvedaemon, pveproxy and pvestatd
3. Start pve-cluster
4. Wait for quorum
5. Delete /etc/pve/authkey.pub and /etc/pve/authkey.old
6. Start pvedaemon, pveproxy and pvestatd

Hope this can help any future Proxmoxers who might face the same issue.

 

[muecke63]

Resolved!

Had to, in a specific order combined with luck (?) do the following:

1. Stop corosync and pve-cluster
2. Stop pvedaemon, pveproxy and pvestatd
3. Start pve-cluster
4. Wait for quorum
5. Delete /etc/pve/authkey.pub and /etc/pve/authkey.old
6. Start pvedaemon, pveproxy and pvestatd

Hope this can help any future Proxmoxers who might face the same issue.

Hello Johan,

thanks for sharing the solution, i have the same behaviour.
2 Node Cluster with external qdevice, everythings up and running pvecm status shows 3 Votes and expected flags.
However if i shutdown node1 for maintenance i can not login to node2 until doing the workaround.

Do you have the same problems within maintenance?

Best regards

Markus

 

[JohanKarlsson]

Hi Markus,

No, I do not face that issue. Sadly I don't have any suggestion as to what could be behind your issue.

Sincerely,
Johan

 

[muecke63]

It seems to i was to fast with trying to login, everythings works now.