Problem with cluster (Adding new node to existing cluster)

B

Binksy

Member
Feb 18, 2023
5
0
6
Hey,

I am unable to get a new node to join a cluster, either via UI or terminal/ssh

Tried a few reinstalls of Proxmox, and ways of getting it to join

Fresh Proxmox 8.2 installation on new Node
8.1.3 on existing node (unable to upgrade at the moment - the node I need to join is supposed to be taking the load as a hot backup to enable downtime on existing nodes)
Existing cluster has 4 nodes already

via UI it gets stuck on "Request addition of this node"

via SSH it gets stuck on "waiting for quorum"

Both times I get booted out of the UI, and unable to reconnect ('authkey' error: no quorum!)
- I can connect via SSH

Servers are in the same datacenter

The node shows up in the cluster, but as disconnected. If I click into it I get

Code: 
hostname lookup '[REDACTED]' failed - failed to get address info for: [REDACTED]: Name or service not known (500)

journalctl -b -u pveproxy -u pvedaemon -u pve-cluster -u corosync

Code: 
Jun 15 10:33:49 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:33:49 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a5f) was formed. Members
Jun 15 10:33:49 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:33:49 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:00 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:00 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a63) was formed. Members
Jun 15 10:34:00 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:00 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:11 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:11 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a67) was formed. Members
Jun 15 10:34:11 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:11 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:22 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:22 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a6b) was formed. Members
Jun 15 10:34:23 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:23 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:33 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:33 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a6f) was formed. Members
Jun 15 10:34:34 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:34 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:34 173-231-63-2 pmxcfs[1726]: [status] notice: cpg_send_message retried 2 times
Jun 15 10:34:45 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:45 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a73) was formed. Members
Jun 15 10:34:45 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:45 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:56 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:56 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a77) was formed. Members
Jun 15 10:34:56 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:56 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:02 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:02 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a7b) was formed. Members
Jun 15 10:35:02 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:02 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:13 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:13 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a7f) was formed. Members
Jun 15 10:35:13 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:13 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:13 173-231-63-2 pmxcfs[1726]: [status] notice: cpg_send_message retried 4 times
Jun 15 10:35:24 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:24 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a83) was formed. Members
Jun 15 10:35:25 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:25 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:35 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:35 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a87) was formed. Members
Jun 15 10:35:36 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:36 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:46 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:46 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a8b) was formed. Members
Jun 15 10:35:47 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:47 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:58 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:58 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a8f) was formed. Members
Jun 15 10:35:58 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:58 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.

pvecm status
On node already in cluster - not showing
Code: 
Cluster information
-------------------
Name:             [REDACTED]
Config Version:   19
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sat Jun 15 10:29:01 2024
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000007
Ring ID:          2.139f7
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      4
Quorum:           3 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 [REDACTED]
0x00000005          1 [REDACTED]
0x00000006          1 [REDACTED]
0x00000007          1 [REDACTED] (local)

On joining node - Cluster name matches
Code: 
Cluster information
-------------------
Name:             [REDACTED]
Config Version:   19
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sat Jun 15 10:30:49 2024
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1.139ab
Quorate:          No

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      1
Quorum:           3 Activity blocked
Flags:           

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 [REDACTED] (local)

journalctl -b -u pveproxy -u pvedaemon -u pve-cluster -u corosync

Code: 
Jun 15 10:33:32 192-69-221-162 corosync[2182]:   [TOTEM ] A new membership (1.13a57) was formed. Members
Jun 15 10:33:32 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 30
Jun 15 10:33:33 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 40
Jun 15 10:33:34 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 50
Jun 15 10:33:35 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 60
Jun 15 10:33:36 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 70
Jun 15 10:33:37 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 80
Jun 15 10:33:38 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 90
Jun 15 10:33:39 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 100
Jun 15 10:33:39 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retried 100 times
Jun 15 10:33:39 192-69-221-162 pmxcfs[2184]: [status] crit: cpg_send_message failed: 6
Jun 15 10:33:40 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 10
Jun 15 10:33:41 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 20
Jun 15 10:33:42 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 30
Jun 15 10:33:43 192-69-221-162 corosync[2182]:   [QUORUM] Sync members[1]: 1
Jun 15 10:33:43 192-69-221-162 corosync[2182]:   [TOTEM ] A new membership (1.13a5b) was formed. Members
Jun 15 10:33:43 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 40
Jun 15 10:33:44 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 50
Jun 15 10:33:45 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 60
Jun 15 10:33:46 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 70

Not sure if any more information is needed to help, but any help is appreciated!

Thanks!
 
You must log in or register to reply here.
Top Bottom