[SOLVED] Adding new node to cluster stalls wating for quorum

[Andres Arias]

Hi everyone!

I'm trying to add a new node(s) to a cluster. I've installed the new servers, have done "apt-get update/dist-upgrade", configured the network on them using openvswitch, tested it is working as expected and then tried to add the new node to the cluster:

root@fuen-vm-2:~# pvecm add 10.0.3.53
copy corosync auth key
stopping pve-cluster service
backup old database
waiting for quorum...

And it stall on quorum... On the existing nodes, from web the new node appears as red.

From the cli, the old nodes shows:

root@fuen-devvm-1:~# pvecm status
Quorum information
------------------
Date: Wed Nov 16 19:32:34 2016
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000003
Ring ID: 1/5540
Quorate: Yes

Votequorum information
----------------------
Expected votes: 5
Highest expected: 5
Total votes: 3
Quorum: 3
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.0.3.47
0x00000002 1 10.0.3.48
0x00000003 1 10.0.3.53 (local)
root@fuen-devvm-1:~# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
1 1 fuen-netvm-1
2 1 fuen-netvm-2
3 1 fuen-devvm-1 (local)
root@fuen-devvm-1:~#

And on the new node:

root@fuen-vm-2:~# pvecm status
Quorum information
------------------
Date: Wed Nov 16 19:34:35 2016
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000005
Ring ID: 5/4
Quorate: No

Votequorum information
----------------------
Expected votes: 5
Highest expected: 5
Total votes: 1
Quorum: 3 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000005 1 10.0.3.43 (local)
root@fuen-vm-2:~# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
5 1 fuen-vm-2 (local)

Any idea on how to proceed? I've tried to add another node using a different member of the cluster and the result is the same!

Please, help!

 

[Andres Arias]

I've found a problem!

So, for all of you who may have similar, issue, try to follow this "Troubleshooting multicast, quorum and cluster issues" guide. I've found that suddenly the multicast has stopped working, and this is why there was no quorum. Investigating further, I've realized that we have had changed our Cisco switches firmware, which lead me to this guide: "Multicast notes". In the new FW version the way of configuring "igmp snooping querier" changes, and once found the problem it was just a matter of finding how it should be done now.