[SOLVED] Help! My cluster node went MIA

proxwolfe

Well-Known Member
Jun 20, 2020
561
69
68
51
Hi,

I have a three node home lab cluster with separate corosync and management networks. One of the nodes was recently replaced with new hardware. When joining the existing cluster, apparently, I used the management network address for the replaced node instead of its corosync network address. That did not pose a problem. Until I realized it and wanted to correct it...

So I edited the corosync.conf to change the new node's ring0_addr from its management network ip to its corosync network ip and increased the config version number. I restarted corosync on the new node and it went dark.

Now, in the GUI, in datacenter > cluster, all three nodes are listed with their corosync IPs (so the change came through). The corosync.conf on all three nodes lists the corosync IP for all nodes. All nodes can ping each other under their corosync IPs.

BUT pvcem status on the old cluster nodes lists only the two old cluster nodes whereas pvecm on the new node only lists the new node (but both factions expect three nodes in total; so the old nodes consider themselves as having quorum wheres the new node considers itself as blocked). The GUI accessed from the old nodes shows the new node with a red X (but I can access its shell via the GUI). (I can't access the GUI on the new node, because it does not accept the MFA configured for the cluster.)

While typing this, I noticed one thing - I don't know whether that might have to do with my problem: pvecm status gives me different Ring IDs on the old nodes vs on the new node. Could

How can I get my new node to come back to the cluster (apart from potentially going back to its old management ip for corosync purposes)?

Any help appreciated!
 
Got it working again (with help from ChatGPT).

The problem was that while the new config files were distributed, the running corosync processes on the old nodes were still using the old (management) IP pf the new node and, hence, rejecting, the new node's packages on the corosync network because they were coming from the (in their view) incorrect address.

Restarting the corosync processes on both old nodes solved the issue.