I just upgraded all my cluster nodes and rebooted. Upon reboot two of them will not join the cluster.
When I run pvecm status on the ones that are seeing each other I get:
Quorum information
------------------
Date: Sun May 22 21:50:10 2016
Quorum provider: corosync_votequorum
Nodes: 4
Node ID: 0x00000004
Ring ID: 28712
Quorate: Yes
Votequorum information
----------------------
Expected votes: 6
Highest expected: 6
Total votes: 4
Quorum: 4
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.1.1.50
0x00000004 1 10.1.1.51 (local)
0x00000003 1 10.1.1.52
0x00000002 1 10.1.1.54
When I run pvecm status on the ones that aren't seeing each other I get:
Quorum information
------------------
Date: Sun May 22 21:49:33 2016
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000005
Ring ID: 26872
Quorate: No
Votequorum information
----------------------
Expected votes: 6
Highest expected: 6
Total votes: 1
Quorum: 4 Activity blocked
Flags:
Membership information
----------------------
Nodeid Votes Name
0x00000005 1 10.1.1.53 (local)
I can ssh and ping from all nodes to each other (including the ones that are not joining the cluster). I have restarted the cluster on the nodes that can't connect via: /etc/init.d/pve-cluster restart to no avail with no error. I did check the syslog and got the following error i have never seen before: Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Any thoughts or ideas would be greatly appreciated.
Thank you,
Ben
When I run pvecm status on the ones that are seeing each other I get:
Quorum information
------------------
Date: Sun May 22 21:50:10 2016
Quorum provider: corosync_votequorum
Nodes: 4
Node ID: 0x00000004
Ring ID: 28712
Quorate: Yes
Votequorum information
----------------------
Expected votes: 6
Highest expected: 6
Total votes: 4
Quorum: 4
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.1.1.50
0x00000004 1 10.1.1.51 (local)
0x00000003 1 10.1.1.52
0x00000002 1 10.1.1.54
When I run pvecm status on the ones that aren't seeing each other I get:
Quorum information
------------------
Date: Sun May 22 21:49:33 2016
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000005
Ring ID: 26872
Quorate: No
Votequorum information
----------------------
Expected votes: 6
Highest expected: 6
Total votes: 1
Quorum: 4 Activity blocked
Flags:
Membership information
----------------------
Nodeid Votes Name
0x00000005 1 10.1.1.53 (local)
I can ssh and ping from all nodes to each other (including the ones that are not joining the cluster). I have restarted the cluster on the nodes that can't connect via: /etc/init.d/pve-cluster restart to no avail with no error. I did check the syslog and got the following error i have never seen before: Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Any thoughts or ideas would be greatly appreciated.
Thank you,
Ben