[SOLVED] Failed to join cluster

[degan]

Hey all,

we have big trouble to join an 4th node in our running 3 node cluster.

PVE Version on all Nodes: 6.3-3

Spoiler: pvecm status on 4th node

Cluster information
-------------------
Name: pve-cluster1
Config Version: 8
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Tue Jan 26 14:01:48 2021
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000004
Ring ID: 4.a79a
Quorate: No

Votequorum information
----------------------
Expected votes: 4
Highest expected: 4
Total votes: 1
Quorum: 3 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000004 1 10.0.20.6 (local)

Spoiler: pvecm status on cluster node

Cluster information
-------------------
Name: pve-cluster1
Config Version: 8
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Tue Jan 26 14:05:01 2021
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000002
Ring ID: 1.a8d6
Quorate: Yes

Votequorum information
----------------------
Expected votes: 4
Highest expected: 4
Total votes: 3
Quorum: 3
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.0.20.8
0x00000002 1 10.0.20.7 (local)
0x00000003 1 10.0.20.13

The 4th node gives me several errors:

Spoiler: pveproxy.service

Jan 26 14:06:09 srv2001 pveproxy[17761]: worker exit
Jan 26 14:06:09 srv2001 pveproxy[17767]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1775.
Jan 26 14:06:09 srv2001 pveproxy[1233]: worker 17760 finished
Jan 26 14:06:09 srv2001 pveproxy[1233]: starting 1 worker(s)
Jan 26 14:06:09 srv2001 pveproxy[1233]: worker 17768 started
Jan 26 14:06:09 srv2001 pveproxy[1233]: worker 17761 finished
Jan 26 14:06:09 srv2001 pveproxy[1233]: starting 1 worker(s)
Jan 26 14:06:09 srv2001 pveproxy[17768]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1775.
Jan 26 14:06:09 srv2001 pveproxy[1233]: worker 17769 started
Jan 26 14:06:09 srv2001 pveproxy[17769]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1775.

Spoiler: pve-cluster.service

Jan 26 14:07:08 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retry 90
Jan 26 14:07:09 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retry 100
Jan 26 14:07:09 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retried 100 times
Jan 26 14:07:09 srv2001 pmxcfs[1055]: [status] crit: cpg_send_message failed: 6
Jan 26 14:07:10 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retry 10
Jan 26 14:07:11 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retry 20
Jan 26 14:07:12 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retry 30
Jan 26 14:07:13 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retry 40
Jan 26 14:07:14 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retry 50
Jan 26 14:07:15 srv2001 pmxcfs[1055]: [status] notice: cpg_send_message retry 60

In cluster view i see the 4th node as offline. If i try to access the settings via gui i get following error:

hostname lookup '<hostname 4th node> failed - failed to get address info for: <hostname 4th node>: Name or service is unknown (500)

I double checked network configuration. Each other host is pingable from 4th node. 4th node is reachable by the other nodes too. We have 2 seperated cluster networks.

I saw that the /etc/pve directory is not fully existing:


Please help us. If you need more information let me know.

 

[fiona]

Hi,
can you ping the nodes also via hostname instead of IPs? Please check that your /etc/hosts information is correct and matches the addresses from the corosync.conf.

 

[degan]

Hi,
can you ping the nodes also via hostname instead of IPs? Please check that your /etc/hosts information is correct and matches the addresses from the corosync.conf.

Hi,
yes, that worked.

We reinstalled the node, cleaned up the cluster and rejoined again.
After this the problem was solved.
Sry for my late reply.