Adding node to cluster causes rsa errors in syslog, nodes see each other as offline

[h9k]

Hi there!


I have a little problem adding a node to the cluster.
Everything went apparently fine in the adding process:

copy corosync auth key
stopping pve-cluster service
Stopping pve cluster filesystem: pve-cluster.
backup old database
Starting pve cluster filesystem : pve-cluster.
Starting cluster:
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Unfencing self... [  OK  ]
waiting for quorum...OK
generating node certificates
merge known_hosts file
restart services
Restarting PVE Daemon: pvedaemon.
Restarting PVE API Proxy Server: pveproxy.
successfully added node 'cosmos' to cluster.

But the situation is, on the two nodes of the cluster, the respective other node appears in the Datacenter tree but is marked as being offline.
So I looked at syslog on the newly added node, and it is full of these messages:

Jun  4 18:13:07 cosmos pveproxy[37553]: problem with client 172.16.0.1; rsa_eay_public_decrypt: data too large for modulus
Jun  4 18:13:08 cosmos pveproxy[37553]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.

And the existing "old" node has these messages:

Jun  4 18:13:17 nebula pveproxy[614653]: problem with client 172.16.0.2; rsa_padding_check_pkcs1_type_1: block type is not 01
Jun  4 18:13:18 nebula pveproxy[646876]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.

Both machines have the latest debian 7 packages with proxmox 3.4 and identical hardware, except for disk drives. Nodes are connected to each other via dedicated nics over vlan.
Pinging and sshing from one machine to the other works flawlessly.
So clearly some kind of rsa problem, something with some certificates? but which ones? Both nodes have the same http wildcard certificates, which work fine in the web browsers. But I am not sure how pveproxy is communicating with other nodes.
Oh and I deleted all cookies and whatnot... but that client IP is the node IP, so it is not a problem with my browser, but with the connection between those two nodes.

 

[h9k]

Re: Adding node to cluster causes rsa errors in syslog, nodes see each other as offli

Just a small update: I tried using another, previously unused IP for the new node, changing hostname, regenerated ssh keys on both nodes. Clean installing Proxmox on the new node. Cleared cache/cookies etc. from all browsers. No avail, still the same problems.

Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] CLM CONFIGURATION CHANGE
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] New Configuration:
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] #011r(0) ip(172.16.0.1)
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] Members Left:
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] Members Joined:
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] CLM CONFIGURATION CHANGE
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] New Configuration:
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] #011r(0) ip(172.16.0.1)
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] Members Left:
Jun  7 16:33:17 nebula corosync[3565]:   [CLM   ] Members Joined:
Jun  7 16:33:17 nebula corosync[3565]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jun  7 16:33:17 nebula corosync[3565]:   [CPG   ] chosen downlist: sender r(0) ip(172.16.0.1) ; members(old:1 left:0)
Jun  7 16:33:17 nebula corosync[3565]:   [MAIN  ] Completed service synchronization, ready to provide service.

These logs also keep repeating on the "primary" node.