Cluster Issues Expanding from 2 to 3 nodes

[lweidig]

We have been for a long time running a two cluster node system and working great. We needed to expand things so we added another node and attempted to join the new box to the cluster. This seems to have worked "partially" but we definitely do not have functionality needed. For example if we go to the web GUI all three nodes are displayed and if I go to the new nodes "Summary" tab the CPU usage, server load, ... graphs are shown but under the status it just sits with "communiction failure (0)" and spins. Clicking on the Services tab will spin loading for quite a while and then give me a "Connection timed out (596)" error ,... This is running version 3.4-6/102d4547 and all nodes have the latest updates applied. Looking for suggestions how to get this fully functional.

New Node:

# pvecm status
Version: 6.2.0
Config Version: 5
Cluster Name: Milwaukee-00
Cluster Id: 49372
Cluster Member: Yes
Cluster Generation: 1784
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2  
Active subsystems: 5
Flags: 
Ports Bound: 0  
Node name: cor-vst-03
Node ID: 3
Multicast addresses: 239.192.192.157 
Node addresses: 172.16.224.36 
root@cor-vst-03:~# pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M   1784   2015-09-09 10:20:16  mke-vst-00
   2   M   1784   2015-09-09 10:20:16  mke-vst-01
   3   M   1780   2015-09-09 10:20:16  cor-vst-03

Old Node:

root@mke-vst-00:~# pvecm status
Version: 6.2.0
Config Version: 5
Cluster Name: Milwaukee-00
Cluster Id: 49372
Cluster Member: Yes
Cluster Generation: 1784
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2  
Active subsystems: 5
Flags: 
Ports Bound: 0  
Node name: mke-vst-00
Node ID: 1
Multicast addresses: 239.192.192.157 
Node addresses: 172.16.224.33 
root@mke-vst-00:~# pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M   1764   2015-09-01 01:51:04  mke-vst-00
   2   M   1764   2015-09-01 01:51:04  mke-vst-01
   3   M   1784   2015-09-09 10:20:17  cor-vst-03

Shared cluster.conf file:

<?xml version="1.0"?>
<cluster name="Milwaukee-00" config_version="5">


  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>


  <clusternodes>
  <clusternode name="mke-vst-00" votes="1" nodeid="1"/>
  <clusternode name="mke-vst-01" votes="1" nodeid="2"/><clusternode name="cor-vst-03" votes="1" nodeid="3"/></clusternodes>


</cluster>

The thing I notice is that for the pvecm nodes command it shows different inc numbers between the nodes and even differen inc numbers from the status for the new node and the old node.

 

[dietmar]

Is there some kind of firewall between the nodes which blocks port 8006?

 

[lweidig]

There is no firewall, but I cannot access 8006 even directly to the new node. I tried running pvecm updatecerts --force and received the following:

# pvecm updatecerts --force
Signature ok
subject=/OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=cor-vst-03.excel.net
Getting CA Private Key
CA certificate and CA private key do not match
139892674999976:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:x509_cmp.c:330:
unable to generate pve ssl certificate:
command 'openssl x509 -req -in /tmp/pvecertreq-6342.tmp -days 3650 -out /etc/pve/nodes/cor-vst-03/pve-ssl.pem -CAkey /etc/pve/priv/pve-root-ca.key -CA /etc/pve/pve-root-ca.pem -CAserial /etc/pve/priv/pve-root-ca.srl -extfile /tmp/pvesslconf-6342.tmp' failed: exit code 1

This is a fresh ISO install with updates applied and that is about it.

 

[lweidig]

This ended up being an issue with the fact that we are using a wildcard certificate for services so that everything is trusted when you access the web UI. Ended up copying the key / pem from one of the original nodes over to the new node /etc/pve/local/pve-ssl.key and pve-ssl.pem files and away it went. Everything is working now.