3 Node cluster "permission denied - invalid PVE ticket (401)"

[BugProgrammer]

Tried to create a 3 node cluster with a fresh proxmox ve 6.0-4 install.
Cluster creation works and adding a second node works aswell, but after i added the 3rd node i get "permission denied - invalid PVE ticket (401)" (only for the third the other 2 are still working).

In the webinterface i can access Node 1 and 2, but 3 aborts with this message. Node 3 can't access any node.

 

[Dominic]

Did you try clearing your browser cache or using a different browser?

 

[BugProgrammer]

Did you try clearing your browser cache or using a different browser?

yes to both


What i tried until now:
-use another browser/workstation to access
-separate the 3rd node and use delnode on the other clients then readd
-tried the above and before readd i cleared all reverences i could find on the 2 working nodes
-checked timedatectl and synced the time and timezone between all nodes
-reinstalled node 3 & synced the time and added it to the cluster again (before i cleared all references from the other nodes)

Nothing of this worked. After "pvecm add ip-of-the-first-node" it says successful and the webpanel shows the node in the cluster with it's local and local lvm. When i expand it i get "permission denied - invalid PVE ticket (401)"...

No idea what i should try next.

 

[BugProgrammer]

crazy, reinstalled all 3 nodes and now it worked

 

[hibouambigu]

Same thing is happening to me too. Fourth cluster I've built, but first time using the GUI and separate corosync network to do so (now with 6.0.4)

Hosts can all ping one-another on corosync network, and all went fine until joining node #2 and #3 via GUI.

Is the corosync cluster network supposed to be able to reach the NTP server directly from that separate network?

EDIT: more detail:

2/3 nodes seem to be ok. The 3rd node has joined the cluster and is visible in the other 2 nodes management windows via web UI.

Node 3 asks for login each time it is visited. Nothing works from this node's web UI, but it does believe it is joined to the cluster (node 1 and 2 are visible, but clicking anything throws errors...401: no ticket in shell, and "NaN" repeatedly in other fields within the cluster management).

 

[hibouambigu]

Same thing is happening to me too. Fourth cluster I've built, but first time using the GUI and separate corosync network to do so (now with 6.0.4)

Hosts can all ping one-another on corosync network, and all went fine until joining node #2 and #3 via GUI.

Is the corosync cluster network supposed to be able to reach the NTP server directly from that separate network?

EDIT: more detail:

2/3 nodes seem to be ok. The 3rd node has joined the cluster and is visible in the other 2 nodes management windows via web UI.

Node 3 asks for login each time it is visited. Nothing works from this node's web UI, but it does believe it is joined to the cluster (node 1 and 2 are visible, but clicking anything throws errors...401: no ticket in shell, and "NaN" repeatedly in other fields within the cluster management).

For anyone else knocking about with this...
Seem to have solved it for now. Still not sure why the error happened during cluster creation!

1.)

pvecm updatecerts
systemctl restart pvedaemon pveproxy

2.) restarted nodes.
3.) cleared browser cookies for all three nodes.

..still had the errors, until the web browser itself was purged of cache, closed and restarted.

 

[Jean-François Dagenais]

Browser doesn't seem to be the issue. I am still trying to fix this. It reoccurs for us every 10 days or so.

 

[gbecker]

Browser doesn't seem to be the issue. I am still trying to fix this. It reoccurs for us every 10 days or so.

OMG, i am not the only one ! Got a cluster with 3 nodes and the error came randomly, sometimes 2/3 days sometimes more.

 

[skywyw]

Helllo i was haveing the same problem the way i fixed it is :
1. Deleting this files:
<node> is your node name

• /etc/pve/pve-root-ca.pem
• /etc/pve/priv/pve-root-ca.key
• /etc/pve/nodes/<node>/pve-ssl.pem
• /etc/pve/nodes/<node>/pve-ssl.key
• /etc/pve/authkey.pub
• /etc/pve/priv/authkey.key
• /etc/pve/priv/authorized_keys

2. pvecm updatecerts -f
3 systemctl restart pvedaemon pveproxy

Hope it works for others too

 

[spirit]

Please also verify that your host clocks are synced

 

[Jean-François Dagenais]

We have a 3 node setup. The lesser 3rd node, only used only as a replication target for backup puporses, still has the issue.

We have since purchased the license subscription and are currently running Virtual Environment 6.1-8. Since the affected node can be rebooted, I have added this as a daily cron job and the problem is worked around this way. I have disabled the reboot last week and today, the node is not reachable from the others with error "permission denied - invalid PVE ticket (401)".

Proxmox, fix this.

 

[Jean-François Dagenais]

My clocks are in sync... when I observed them.

This could be a good clue still. My 2 main nodes are bare-metal, but my 3rd node is a VM (bhyve). Maybe the host's network timesync is messing with the date periodically? Anyone can put some weight on this?

 

[elmo]

My clocks are in sync... when I observed them.

This could be a good clue still. My 2 main nodes are bare-metal, but my 3rd node is a VM (bhyve). Maybe the host's network timesync is messing with the date periodically? Anyone can put some weight on this?

A late reply perhaps, but I think you might be on to something, as I've come across this before. In many cases the default value for a VM is to get its
time synced with the parent partition, eg. from the host it's running on. Make sure this is not the case for you 3rd node and that all of your hosts are
using the same time source.

 

[serafin.rusu]

I solved this by setting up the same NTP server on all servers.

 

[Kaijia]

Helllo i was haveing the same problem the way i fixed it is :
1. Deleting this files:
<node> is your node name

• /etc/pve/pve-root-ca.pem
• /etc/pve/priv/pve-root-ca.key
• /etc/pve/nodes/<node>/pve-ssl.pem
• /etc/pve/nodes/<node>/pve-ssl.key
• /etc/pve/authkey.pub
• /etc/pve/priv/authkey.key
• /etc/pve/priv/authorized_keys

2. pvecm updatecerts -f
3 systemctl restart pvedaemon pveproxy

Hope it works for others too

Can confirm this works. My cluster had this 401 issue on all nodes (not just one), I had tried ntp and pvecm updatecerts and reboot the whole cluster but all failed. I ended up fixing this using this method, and replace pve-ssl cert on all nodes. Thanks skywyw.

 

[j4ys0n]

Helllo i was haveing the same problem the way i fixed it is :
1. Deleting this files:
<node> is your node name

• /etc/pve/pve-root-ca.pem
• /etc/pve/priv/pve-root-ca.key
• /etc/pve/nodes/<node>/pve-ssl.pem
• /etc/pve/nodes/<node>/pve-ssl.key
• /etc/pve/authkey.pub
• /etc/pve/priv/authkey.key
• /etc/pve/priv/authorized_keys

2. pvecm updatecerts -f
3 systemctl restart pvedaemon pveproxy

Hope it works for others too

@skywyw which node(s) should i run these commands on?

i have a 4 node cluster and 1 is giving me the "permission denied - invalid PVE ticket (401)" error.

and do i remove pve-ssl.pem & pve-ssl.key for just the one that's having trouble or all nodes?

 

[bgirgis]

I had the same problem and it turned out the new node had a faulty DNS server entry. Fixing that resolved the issue.

 

[blazek]

@skywyw which node(s) should i run these commands on?

i have a 4 node cluster and 1 is giving me the "permission denied - invalid PVE ticket (401)" error.

and do i remove pve-ssl.pem & pve-ssl.key for just the one that's having trouble or all nodes?

I have similar problem. 5 nodes and only one is giving me the "permission denied - invalid PVE ticket (401)" error.
Have you any solution? I tried set up same ntp server on all the nodes - did not help.

It's production servers, so I cant reboot them as it would suit me.

 

[kazemian]

I solved this by setting up the same NTP server on all servers.

thanks.its working

 

[PMDS]

Hi

I know this is a rather old thread but it might help people come across ... I encountered the same error as mentioned on a fresh installed 3 node Proxmox VE cluster.

When switching from one node to the other in webgui the 401 error came up - as it is a testing cluster which is hibernated from time to time I realized following points:

- after suspending and waking up the machines there may be a time difference and according logfiles some actions do not tolerate a difference of more than one second
- the browser must know about all certificates and have them accepted if using self signed certs (login with all addresses of all nodes)
- browser cache should be cleared
- and storing username / pw may help (but for a production cluster I would not recommend this)

Regards, Dietmar