Failed to add cluster node "401 No ticket"

[n1nj4888]

Can anyone help with the following error I'm getting after creating a cluster on a single node and attempting to add a second node to the cluster?

I've created a cluster on the first PVE Node (call it NODE1) which has the following networks configured:

DefaultLAN - 192.168.1.NODE1
ClusterHB (Ring0 Address) - 192.168.2.NODE1

I also have a second standalone PVE Node (call it NODE2) which has the following networks configured:

DefaultLAN - 192.168.1.NODE2
ClusterHB (Ring0 Address) - 192.168.2.NODE2

When i copy the Cluster Join Information from NODE1 to NODE2 (in the GUI of NODE2), I paste the Join Information into "Join Cluster" and:

1) Peer Address is auto-populated with the DefaultLAN address of NODE1 (i.e. 192.168.1.NODE1 not the NODE1 Ring0 Address - Assume this correct?)
2) Corosync Ring 0 is populated with a warning: "Cannot use default address safely" so I update this to be ClusterHB (Ring 0 Address) of NODE2 i.e. 192.168.2.NODE2
3) Enter the root password of NODE1
4) Click "Join Cluster"

The following error is displayed in the Task Viewer for the "Join Cluster" job?
Establishing API connection with host '192.168.1.NODE1'
Login succeeded.
Request addition of this node
TASK ERROR: 401 No ticket

I also get the same type of error if I try to add NODE2 to the cluster via NODE2 console:

root@NODE2:~# pvecm add 192.168.1.NODE1 -ring0_addr 192.168.2.NODE2
Please enter superuser (root) password for '192.168.1.NODE1':
Password for root@192.168.1.NODE1: ***
Establishing API connection with host '192.168.1.NODE1'
The authenticity of host '192.168.1.NODE1' can't be established.
X509 SHA256 key fingerprint is XXX.
Are you sure you want to continue connecting (yes/no)? yes
Login succeeded.
Request addition of this node
401 No ticket

 

[t.lamprecht]

Can you please post the cluster corosync configuration from the node you created the cluster? E.g.:

cat /etc/pve/corosync.conf

It would be great to have uncensored, as long as those node use private IPv4 address ranges, like 192.168.0.0/16 or 10.0.0.0/8, only.

Can you login over the web interface as root on both nodes normally? What's going on in the syslog/journal during this error?

Also, as workaround - joining over SSH is still available, you could try it by adding the "--use_ssh" to the pvecm add commando.

 

[n1nj4888]

Thanks for the reply!

Here is /etc/pve/corosync.conf from NODE1 (where I created the cluster)

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-host1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.2.201
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: pve-cluster1
  config_version: 1
  interface {
    bindnetaddr: 192.168.2.201
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

Yes - I can login to the WebUI as root on both NODE1 and NODE2 as expected...

The following is the output on NODE2 syslog when attempting to join the cluster through the NODE2 WebUI:

Jun  4 12:30:09 pve-host2.local pvedaemon[29538]: <root@pam> starting task UPID:pve-host2:<ID>:clusterjoin::root@pam:
Jun  4 12:30:12 pve-host2.local pvedaemon[14119]: 401 No ticket
Jun  4 12:30:12 pve-host2.local pvedaemon[29538]: <root@pam> end task UPID:pve-host2:<ID>:clusterjoin::root@pam: 401 No ticket

There doesn't seem to be anything of interest in the NODE1 syslog at the point of attempting to join the cluster from NODE2

 

[n1nj4888]

I didn't try the pvecm "--use_ssh" command but, I've found the cause of the issue...

The reason for the error is seemingly because 2FA was turned on for root on both NODE1 and NODE2. As soon as I disabled 2FA (on both nodes) and repasted the Join Information into NODE2, NODE2 was added to the cluster successfully...

2FA is not mentioned anywhere as being a potential issue in the Cluster Manager wiki (https://pve.proxmox.com/wiki/Cluster_Manager) and it's not immediately clear that this could be an issue when adding a node to the cluster so it would be good to either (A) update the wiki to point out that 2FA needs to be disabled for this to work or (B) allow the user to enter the NODE1 2FA key in the NODE2 Join Cluster dialog when attempting to join NODE2 to the cluster

Thanks!

EDIT: After removing 2FA from both nodes and adding NODE2 to the cluster, I was able to re-enable 2FA on NODE1 and confirmed that the same 2FA key can be used to login to either NODE1 or NODE2

 

[t.lamprecht]

The reason for the error is seemingly because 2FA was turned on for root on both NODE1 and NODE2. As soon as I disabled 2FA (on both nodes) and repasted the Join Information into NODE2, NODE2 was added to the cluster successfully...

Ahh, yeah, now the error makes sense..

2FA is not mentioned anywhere as being a potential issue in the Cluster Manager wiki (https://pve.proxmox.com/wiki/Cluster_Manager) and it's not immediately clear that this could be an issue when adding a node to the cluster so it would be good to either (A) update the wiki to point out that 2FA needs to be disabled for this to work or (B) allow the user to enter the NODE1 2FA key in the NODE2 Join Cluster dialog when attempting to join NODE2 to the cluster

Yes, there we definitively lack some information... The cluster join API wasn't adapted to the new improved TFA integration...
So, for TOTP we could really allow to enter the code if it's set, for U2F it's not to easy to interact with the keys in a manageable way, not sure about that one, here we could at least warn the user on the CLI, for webUI it could be even done...

 

[t.lamprecht]

I opened: https://bugzilla.proxmox.com/show_bug.cgi?id=2227 to track this, thanks for your report here!

 

[bferrell]

Thank you, Lord, for delivering me!... I would likely not have thought of this, but got me over the hump as well...