Problem adding third node to cluster

[dendi]

Hello,
I had a problem adding the third node to the PVE 5.0 cluster, updated with no-subscription repo.
I added node 3 using the IP of node 2.
Node 2 and node 3 see three nodes.
Node 1 sees only node 2.

root@pvez5:~# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
1 1 pvez5 (local)
2 1 pvez6
3 1 pvez7.domain.net <------------- IS THIS THE PROBLEM?


root@pvez6:~# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
1 1 pvez5
2 1 pvez6 (local)
3 1 pvez7
root@pvez7:~# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
1 1 pvez5
2 1 pvez6
3 1 pvez7 (local)

I had nothing in file hosts and the same resolv.conf on all nodes.


pvecm status is ok on all nodes:
Quorum information
------------------
Date: Tue Aug 29 22:54:31 2017
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 1/4296
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.1.5 (local)
0x00000002 1 192.168.1.6
0x00000003 1 192.168.1.7

UPDATE: adding a VM on pvez7 (node 3) from pvez7 interface, pvez7 appeared on pvez5 in "red" state
Of course the two nodes can't communicate each other

How I can solve and how that happened?

Thank you

 

[dendi]

I'm going to power off the new server, remove from cluster and reformatting...
have i to remove pvez7.domain.net or pvez7 withouth domain?
Is there another way?

 

[gosha]

I'm going to power off the new server, remove from cluster and reformatting...
have i to remove pvez7.domain.net or pvez7 withouth domain?
Is there another way?

Compare with /etc/hosts on other nodes.

 

[dendi]

Hi gosha,
The hosts files was the default:

127.0.0.1 localhost.localdomain localhost
192.168.1.5 pvez5.domain.net pvez5 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

127.0.0.1 localhost.localdomain localhost
192.168.1.6 pvez6.domain.net pvez6 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

127.0.0.1 localhost.localdomain localhost
192.168.1.7 pvez7.domain.net pvez7 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

 

[dendi]

Sorry for posting again.
But I'm interested to know how this can happen.
I already created 3-4 cluster with no problems.
I have an idea.
This time I did join the new node using the IP address of the second node.
Other times I always used the IP of the node where I created the cluster.

I see corosync.conf:

totem {
  cluster_name: pvezc1
  config_version: 3
  ip_version: ipv4
  secauth: on
  version: 2
  interface {
    bindnetaddr: 192.168.1.5 (ip of the node where the cluster was created)
    ringnumber: 0
  }

}

So we need to use that IP when adding nodes?

 

[gosha]

Your hosts file (on each node!) must contain information about all nodes in cluster!
For example:

192.168.1.6 pvez6.domain.net pvez6 pvelocalhost
192.168.1.7 pvez7.domain.net pvez7
192.168.1.5 pvez5.domain.net pvez5
and so on...

Best regards,
Gosha

 

[Digitaldaz]

Your hosts file (on each node!) must contain information about all nodes in cluster!
For example:

192.168.1.6 pvez6.domain.net pvez6 pvelocalhost
192.168.1.7 pvez7.domain.net pvez7
192.168.1.5 pvez5.domain.net pvez5
and so on...

Best regards,
Gosha

That isn't right, I have a pefectly working cluster with the same hosts as the guy above, in older versions you used to.

 

[dendi]

anyway I put all nodes in all hosts files.
meantime cluster begun irresponsive because of /etc/pve filesystem was blocked.
I rebooted all nodes and now it's all ok.
I noticed that omping doesn't work, maybe this te cause?

 

[gosha]

I noticed that omping doesn't work, maybe this te cause?

Yes. See here: https://pve.proxmox.com/wiki/Multicast_notes

 

[dendi]

Omg... I'm sorry... I have to run omping on all the node at the same time.... it's working!

Anyway I reinstall from scratch and created the new cluster with separate network for corosync and with all names in /etc/hosts

Thank you for your support!