Proxmox 6.2 broken cluster

[Tsunami]

Hi guys,

I am having this specific problem, looking through the forum I found pieces that relate to my problem but not the exact same issue.
What I did:

I had a 2 node cluster running, I wanted to join a 3rd node, problem was the 3rd node could not communicate on the replication network.
I pasted the join Information and had a timeout about 5 Minutes later.
I reinstalled the 3rd node with a different hostname (but same IPs)
Now what I have:
My two nodes are running in Standalone node - no cluster defined
Although in the webgui I see the 3rd node as offline. Its not listed in the crush map but I see the IP entries in the corosync.conf.


I wanted to upgrade the server hardware by rotating: Changing one server (Raid configuration and ram upgrade) then reinstalling proxmox and rejoining the cluster.
this way I would have no downtime, at least I thought so.

Any suggestions on how to clean this up?
Would be great to at least go back to a running two node cluster and try to join the 3rd node again.
All nodes see each other now on every network interface.

 

[Moayad]

Any error in syslog during the adding the 3th node?

May you post the network config as well

 

[Tsunami]

Hi,

of course heres the network config of all 3 nodes, I believe the Problem was that during the join the new node could not communicate with the other 2 on the replication network and therefore could not join correctly.
I did not see any errors that would indicate a problem but since then I already reinstalled it as you are not supposed to remove and join the same installation from a cluster?

Node1:
auto lo
iface lo inet loopback

iface enp4s0f0 inet manual

iface enp4s0f1 inet manual

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
address 91.XXX.XXX.90/24
gateway 10.1.255.1
bridge-ports enp4s0f0
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.100.90/23
bridge-ports enp4s0f1
bridge-stp off
bridge-fd 0

auto vmbr2
iface vmbr2 inet static
address 192.168.17.1/24
bridge-ports enp5s0f1
bridge-stp off
bridge-fd 0


Node2:
auto lo
iface lo inet loopback

iface enp4s0f1 inet manual

iface enp4s0f0 inet manual

auto enp5s0f0
iface enp5s0f0 inet manual

auto enp5s0f1
iface enp5s0f1 inet manual

auto bond0
iface bond0 inet static
address 192.168.17.2/24
bond-slaves enp5s0f0 enp5s0f1
bond-miimon 100
bond-mode balance-rr
#replication network

auto vmbr0
iface vmbr0 inet static
address 91.XXX.XXX.91/24
gateway 10.1.255.1
bridge-ports enp4s0f1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.100.91/23
bridge-ports enp4s0f0
bridge-stp off
bridge-fd 0
#Internal


Node 3:

auto lo
iface lo inet loopback

iface enp3s0f0 inet manual

iface enp3s0f1 inet manual

iface enp4s0f0 inet manual

iface enp4s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
address 91.XXX.XXX.93/24
gateway 10.1.255.1
bridge-ports enp3s0f0
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.100.92/23
bridge-ports enp3s0f1
bridge-stp off
bridge-fd 0

auto vmbr2
iface vmbr2 inet static
address 192.168.17.3/24
bridge-ports enp4s0f1
bridge-stp off
bridge-fd 0