Adding node kills cluster

[coreyjohnson75]

I have an existing year-old 4 node cluster, 6.2-6, but each time I try to add a newly provisioned 5tg node, it kills the existing cluster and causes 2 other nodes to reboot.

I have not doubt that this is something I am doing wrong but have no idea what it is. I install and get the new node running, updates & NTP, use the GUI to join the existing cluster (how I did the others) and then lose everything. If I power off the new node or at least remove all the network connections, the initial 4 work fine.

**Update, to make things more interesting, I booted it without a network connection and then later plugged it in. When I did it populates but will not let be see anything in it, I cannot access it directly, and the rest of the cluster thinks it is no longer a cluster:

 

[blackpaw]

Hmmm, I had the same problem recently (couple weeks ago) - adding two new nodes, both killed the cluster and rebooted everything. Nodes where joined ok after, though I hade to run update certs on them to do anything.

At the time I put it down to the cluster being under load and corosync failing.

 

[coreyjohnson75]

Thanks for responding. Not sure what it was but never resolved after multiple reboot. The more I troubleshooted and Googled, the worse it got (or I made it). Lost all storage connections, even local node LVM?
This is production system so I've been working all night to build/rebuild a new cluster. Most of the way there and no issues.

 

[blackpaw]

Oh man, I feel your pain Sorry that I don't have more to add. Probably too late to debug now anyway.

Good luck!