Joining Node after Cluster Quorum Issues Causes More Networking Issues

[ev0lution37]

Hi all,

I had some network funkiness the other day and long story short, I lost quorum in my 5 node cluster. I got it back for all but one node, so I have a healthy 4 node (prox1-prox4) cluster right now. The one problem node, since there wasn't anything critical on it, I decided to reformat with VE 8.2-2 and rejoin the cluster as 'prox5'. When that cluster comes on-line initially as a single-node cluster, it's perfectly happy. However, when I attempt to join it to the other 4 nodes:

• On one of the existing nodes (prox1), the new node "appears" with 'pvesh get /cluster/status', however it never gets an IP in the list.
• One of the other existing nodes (prox4) _goes offline_ and also loses its IP in that list. If I SSH into that node and try 'pvesh get /cluster/status', it hangs indefinitely. The other 3 nodes (prox1-prox3) stay healthy.
• I can no long SSH into the new node (prox5) that I'm trying to join. SSH hangs indefinitely. If I access it via PiKVM and run 'pvesh get /cluster/status', the same things happens as prox4, it hangs.
• If I unplug the new node, everything on the other 4 nodes goes back to the healthy, including prox4. I get right back to a content cluster, with my new node (prox5) showing as offline.

I'm assuming I'm somehow in quorum hell, but I really don't know where to start. I'm using an entirely different hostname and IP for the node I'm trying to join, but it is the same physical piece of hardware.

NOTE: When I cleaned up the old node before reformatting it, I did:

pvecm delnode prox5
rm -rf /etc/pve/nodes/prox5

Any help/ideas would be much appreciated. Thank you!