Failed to add cluster node "401 No ticket"

[n1nj4888]

Can anyone help with the following error I'm getting after creating a cluster on a single node and attempting to add a second node to the cluster?

I've created a cluster on the first PVE Node (call it NODE1) which has the following networks configured:

DefaultLAN - 192.168.1.NODE1
ClusterHB (Ring0 Address) - 192.168.2.NODE1

I also have a second standalone PVE Node (call it NODE2) which has the following networks configured:

DefaultLAN - 192.168.1.NODE2
ClusterHB (Ring0 Address) - 192.168.2.NODE2

When i copy the Cluster Join Information from NODE1 to NODE2 (in the GUI of NODE2), I paste the Join Information into "Join Cluster" and:

1) Peer Address is auto-populated with the DefaultLAN address of NODE1 (i.e. 192.168.1.NODE1 not the NODE1 Ring0 Address - Assume this correct?)
2) Corosync Ring 0 is populated with a warning: "Cannot use default address safely" so I update this to be ClusterHB (Ring 0 Address) of NODE2 i.e. 192.168.2.NODE2
3) Enter the root password of NODE1
4) Click "Join Cluster"

The following error is displayed in the Task Viewer for the "Join Cluster" job?
Establishing API connection with host '192.168.1.NODE1'
Login succeeded.
Request addition of this node
TASK ERROR: 401 No ticket

I also get the same type of error if I try to add NODE2 to the cluster via NODE2 console:

root@NODE2:~# pvecm add 192.168.1.NODE1 -ring0_addr 192.168.2.NODE2
Please enter superuser (root) password for '192.168.1.NODE1':
Password for root@192.168.1.NODE1: ***
Establishing API connection with host '192.168.1.NODE1'
The authenticity of host '192.168.1.NODE1' can't be established.
X509 SHA256 key fingerprint is XXX.
Are you sure you want to continue connecting (yes/no)? yes
Login succeeded.
Request addition of this node
401 No ticket

 

[t.lamprecht]

Can you please post the cluster corosync configuration from the node you created the cluster? E.g.:

cat /etc/pve/corosync.conf

It would be great to have uncensored, as long as those node use private IPv4 address ranges, like 192.168.0.0/16 or 10.0.0.0/8, only.

Can you login over the web interface as root on both nodes normally? What's going on in the syslog/journal during this error?

Also, as workaround - joining over SSH is still available, you could try it by adding the "--use_ssh" to the pvecm add commando.

 

[n1nj4888]

Thanks for the reply!

Here is /etc/pve/corosync.conf from NODE1 (where I created the cluster)

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-host1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.2.201
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: pve-cluster1
  config_version: 1
  interface {
    bindnetaddr: 192.168.2.201
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

Yes - I can login to the WebUI as root on both NODE1 and NODE2 as expected...

The following is the output on NODE2 syslog when attempting to join the cluster through the NODE2 WebUI:

Jun  4 12:30:09 pve-host2.local pvedaemon[29538]: <root@pam> starting task UPID:pve-host2:<ID>:clusterjoin::root@pam:
Jun  4 12:30:12 pve-host2.local pvedaemon[14119]: 401 No ticket
Jun  4 12:30:12 pve-host2.local pvedaemon[29538]: <root@pam> end task UPID:pve-host2:<ID>:clusterjoin::root@pam: 401 No ticket

There doesn't seem to be anything of interest in the NODE1 syslog at the point of attempting to join the cluster from NODE2

 

[n1nj4888]

I didn't try the pvecm "--use_ssh" command but, I've found the cause of the issue...

The reason for the error is seemingly because 2FA was turned on for root on both NODE1 and NODE2. As soon as I disabled 2FA (on both nodes) and repasted the Join Information into NODE2, NODE2 was added to the cluster successfully...

2FA is not mentioned anywhere as being a potential issue in the Cluster Manager wiki (https://pve.proxmox.com/wiki/Cluster_Manager) and it's not immediately clear that this could be an issue when adding a node to the cluster so it would be good to either (A) update the wiki to point out that 2FA needs to be disabled for this to work or (B) allow the user to enter the NODE1 2FA key in the NODE2 Join Cluster dialog when attempting to join NODE2 to the cluster

Thanks!

EDIT: After removing 2FA from both nodes and adding NODE2 to the cluster, I was able to re-enable 2FA on NODE1 and confirmed that the same 2FA key can be used to login to either NODE1 or NODE2

 

[t.lamprecht]

The reason for the error is seemingly because 2FA was turned on for root on both NODE1 and NODE2. As soon as I disabled 2FA (on both nodes) and repasted the Join Information into NODE2, NODE2 was added to the cluster successfully...

Ahh, yeah, now the error makes sense..

2FA is not mentioned anywhere as being a potential issue in the Cluster Manager wiki (https://pve.proxmox.com/wiki/Cluster_Manager) and it's not immediately clear that this could be an issue when adding a node to the cluster so it would be good to either (A) update the wiki to point out that 2FA needs to be disabled for this to work or (B) allow the user to enter the NODE1 2FA key in the NODE2 Join Cluster dialog when attempting to join NODE2 to the cluster

Yes, there we definitively lack some information... The cluster join API wasn't adapted to the new improved TFA integration...
So, for TOTP we could really allow to enter the code if it's set, for U2F it's not to easy to interact with the keys in a manageable way, not sure about that one, here we could at least warn the user on the CLI, for webUI it could be even done...

 

[t.lamprecht]

I opened: https://bugzilla.proxmox.com/show_bug.cgi?id=2227 to track this, thanks for your report here!

 

[bferrell]

Thank you, Lord, for delivering me!... I would likely not have thought of this, but got me over the hump as well...

 

[Stratus]

I ran into this as well. I removed the TOTP for the root user, but let it for my other user but am still running into that no ticket error. So I removed TOTP completely from all users, but I got the same error. So I tried using the command line / ssh instead, but I got the same error:

1. Node1 is a {PVE} with the hostname bastion
2. Node2 is a {PMG} with the hostname mailgate

root@mailgate:~# pmgcm join 192.168.115.18
cluster join failed: 500 Can't connect to 192.168.115.18:8006 (hostname verification failed)
root@mailgate:~# pmgcm join 192.168.115.18 --fingerprint REA:CT:ED
cluster join failed: 401 No ticket

Node1(bastion) {PVE}: /etc/pve/corosync.conf

logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: bastion
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.115.18
}
}

quorum {
provider: corosync_votequorum
}
totem {
cluster_name: bastion-cluster
config_version: 1
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

Earlier syslog during cluster creation on node 1 (bastion) (PVE)

Dec 02 01:55:09 bastion pvedaemon[161613]: <root@pam> starting task UPID:bastion:00028508:00331805:674D75ED:clustercreate:bastion-cluster:root@pam:
Dec 02 01:55:09 bastion systemd[1]: corosync.service - Corosync Cluster Engine was skipped because of an unmet condition check (ConditionPathExists=/etc/corosync/corosync.conf).
Dec 02 01:55:09 bastion systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem...
Dec 02 01:55:09 bastion pmxcfs[1097]: [main] notice: teardown filesystem
Dec 02 01:55:10 bastion systemd[1]: etc-pve.mount: Deactivated successfully.
Dec 02 01:55:10 bastion pmxcfs[1097]: [main] notice: exit proxmox configuration filesystem (0)
Dec 02 01:55:10 bastion systemd[1]: pve-cluster.service: Deactivated successfully.
Dec 02 01:55:10 bastion systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Dec 02 01:55:10 bastion systemd[1]: pve-cluster.service: Consumed 28.213s CPU time.
Dec 02 01:55:10 bastion systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Dec 02 01:55:10 bastion pmxcfs[165132]: [main] notice: resolved node name 'bastion' to '192.168.115.18' for default node IP address
Dec 02 01:55:10 bastion pmxcfs[165132]: [main] notice: resolved node name 'bastion' to '192.168.115.18' for default node IP address
Dec 02 01:55:10 bastion pmxcfs[165132]: [dcdb] notice: wrote new corosync config '/etc/corosync/corosync.conf' (version = 1)
Dec 02 01:55:10 bastion pmxcfs[165132]: [dcdb] notice: wrote new corosync config '/etc/corosync/corosync.conf' (version = 1)
Dec 02 01:55:10 bastion pmxcfs[165134]: [quorum] crit: quorum_initialize failed: 2
Dec 02 01:55:10 bastion pmxcfs[165134]: [quorum] crit: can't initialize service
Dec 02 01:55:10 bastion pmxcfs[165134]: [confdb] crit: cmap_initialize failed: 2
Dec 02 01:55:10 bastion pmxcfs[165134]: [confdb] crit: can't initialize service
Dec 02 01:55:10 bastion pmxcfs[165134]: [dcdb] crit: cpg_initialize failed: 2
Dec 02 01:55:10 bastion pmxcfs[165134]: [dcdb] crit: can't initialize service
Dec 02 01:55:10 bastion pmxcfs[165134]: [status] crit: cpg_initialize failed: 2
Dec 02 01:55:10 bastion pmxcfs[165134]: [status] crit: can't initialize service
Dec 02 01:55:11 bastion systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
Dec 02 01:55:11 bastion systemd[1]: Starting corosync.service - Corosync Cluster Engine...
Dec 02 01:55:11 bastion pvedaemon[161613]: <root@pam> end task UPID:bastion:00028508:00331805:674D75ED:clustercreate:bastion-cluster:root@pam: OK
Dec 02 01:55:11 bastion corosync[165140]: [MAIN ] Corosync Cluster Engine starting up
Dec 02 01:55:11 bastion corosync[165140]: [MAIN ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf vqsim nozzle snmp pie relro bindnow
Dec 02 01:55:11 bastion corosync[165140]: [TOTEM ] Initializing transport (Kronosnet).
Dec 02 01:55:12 bastion kernel: sctp: Hash tables configured (bind 1024/1024)
Dec 02 01:55:12 bastion corosync[165140]: [TOTEM ] totemknet initialized
Dec 02 01:55:12 bastion corosync[165140]: [KNET ] pmtud: MTU manually set to: 0
Dec 02 01:55:12 bastion corosync[165140]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Dec 02 01:55:12 bastion corosync[165140]: [SERV ] Service engine loaded: corosync configuration map access [0]
Dec 02 01:55:12 bastion corosync[165140]: [QB ] server name: cmap
Dec 02 01:55:12 bastion corosync[165140]: [SERV ] Service engine loaded: corosync configuration service [1]
Dec 02 01:55:12 bastion corosync[165140]: [QB ] server name: cfg
Dec 02 01:55:12 bastion corosync[165140]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Dec 02 01:55:12 bastion corosync[165140]: [QB ] server name: cpg
Dec 02 01:55:12 bastion corosync[165140]: [SERV ] Service engine loaded: corosync profile loading service [4]
Dec 02 01:55:12 bastion corosync[165140]: [SERV ] Service engine loaded: corosync resource monitoring service [6]
Dec 02 01:55:12 bastion corosync[165140]: [WD ] Watchdog not enabled by configuration
Dec 02 01:55:12 bastion corosync[165140]: [WD ] resource load_15min missing a recovery key.
Dec 02 01:55:12 bastion corosync[165140]: [WD ] resource memory_used missing a recovery key.
Dec 02 01:55:12 bastion corosync[165140]: [WD ] no resources configured.
Dec 02 01:55:12 bastion corosync[165140]: [SERV ] Service engine loaded: corosync watchdog service [7]
Dec 02 01:55:12 bastion corosync[165140]: [QUORUM] Using quorum provider corosync_votequorum
Dec 02 01:55:12 bastion corosync[165140]: [QUORUM] This node is within the primary component and will provide service.
Dec 02 01:55:12 bastion corosync[165140]: [QUORUM] Members[0]:
Dec 02 01:55:12 bastion corosync[165140]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Dec 02 01:55:12 bastion corosync[165140]: [QB ] server name: votequorum
Dec 02 01:55:12 bastion corosync[165140]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Dec 02 01:55:12 bastion corosync[165140]: [QB ] server name: quorum
Dec 02 01:55:12 bastion corosync[165140]: [TOTEM ] Configuring link 0
Dec 02 01:55:12 bastion corosync[165140]: [TOTEM ] Configured link number 0: local addr: 192.168.115.18, port=5405
Dec 02 01:55:12 bastion corosync[165140]: [KNET ] link: Resetting MTU for link 0 because host 1 joined
Dec 02 01:55:12 bastion corosync[165140]: [QUORUM] Sync members[1]: 1
Dec 02 01:55:12 bastioncorosync[165140]: [QUORUM] Sync joined[1]: 1
Dec 02 01:55:12 bastion corosync[165140]: [TOTEM ] A new membership (1.5) was formed. Members joined: 1
Dec 02 01:55:12 bastion corosync[165140]: [QUORUM] Members[1]: 1
Dec 02 01:55:12 bastion corosync[165140]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 02 01:55:12 bastion systemd[1]: Started corosync.service - Corosync Cluster Engine.
Dec 02 01:55:15 bastion pvestatd[1167]: storage 'artemis-smb' is not online
Dec 02 01:55:16 bastion pmxcfs[165134]: [status] notice: update cluster info (cluster name bastion-cluster, version = 1)
Dec 02 01:55:16 bastion pmxcfs[165134]: [status] notice: node has quorum
Dec 02 01:55:16 bastion pmxcfs[165134]: [dcdb] notice: members: 1/165134
Dec 02 01:55:16 bastion pmxcfs[165134]: [dcdb] notice: all data is up to date
Dec 02 01:55:16 bastion pmxcfs[165134]: [status] notice: members: 1/165134
Dec 02 01:55:16 bastion pmxcfs[165134]: [status] notice: all data is up to date

node2 (mailgate) {PMG} SysLog

Dec 02 19:41:17 mailgate pmgdaemon[400]: starting task UPID:mailgate:0000205D:0094B34D:674E6FCD:clusterjoin::root@pam:
Dec 02 19:41:17 mailgate pmgdaemon[8285]: TFA-enabled login currently works only with a TTY. at /usr/share/perl5/PVE/APIClient/LWP.pm line 121
Dec 02 19:41:17 mailgate pmgdaemon[400]: end task UPID:mailgate:0000205D:0094B34D:674E6FCD:clusterjoin::root@pam: TFA-enabled login currently works only with a TTY. at /usr/share/perl5/PVE/APIClient/LWP.pm line 121

Node2 (mailgate) {PMG} SYSLOG After removing TFA on node1 (bastion) {PVE}

Dec 02 20:44:35 mailgate pmgdaemon[400]: starting task UPID:mailgate:00004FB8:009A7EE9:674E7EA3:clusterjoin::root@pam:
Dec 02 20:44:38 mailgate pmgdaemon[20408]: 401 No ticket
Dec 02 20:44:38 mailgate pmgdaemon[400]: end task UPID:mailgate:00004FB8:009A7EE9:674E7EA3:clusterjoin::root@pam: 401 No ticket

 

[t.lamprecht]

1. Node1 is a {PVE} with the hostname bastion
2. Node2 is a {PMG} with the hostname mailgate

You cannot join a PMG node with a PVE node. There are PMG clusters, which is an active-standby design, and there is the PVE cluster, which is a multi-master design. Besides completely incompatible cluster stack they are also quite different products, so joining them into a cluster makes not much sense to me.