Problem with cluster (Adding new node to existing cluster)

B

Binksy

New Member
Feb 18, 2023
5
0
1
Hey,

I am unable to get a new node to join a cluster, either via UI or terminal/ssh

Tried a few reinstalls of Proxmox, and ways of getting it to join

Fresh Proxmox 8.2 installation on new Node
8.1.3 on existing node (unable to upgrade at the moment - the node I need to join is supposed to be taking the load as a hot backup to enable downtime on existing nodes)
Existing cluster has 4 nodes already

via UI it gets stuck on "Request addition of this node"

via SSH it gets stuck on "waiting for quorum"

Both times I get booted out of the UI, and unable to reconnect ('authkey' error: no quorum!)
- I can connect via SSH

Servers are in the same datacenter

The node shows up in the cluster, but as disconnected. If I click into it I get

Code: 
hostname lookup '[REDACTED]' failed - failed to get address info for: [REDACTED]: Name or service not known (500)

journalctl -b -u pveproxy -u pvedaemon -u pve-cluster -u corosync

Code: 
Jun 15 10:33:49 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:33:49 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a5f) was formed. Members
Jun 15 10:33:49 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:33:49 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:00 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:00 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a63) was formed. Members
Jun 15 10:34:00 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:00 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:11 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:11 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a67) was formed. Members
Jun 15 10:34:11 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:11 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:22 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:22 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a6b) was formed. Members
Jun 15 10:34:23 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:23 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:33 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:33 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a6f) was formed. Members
Jun 15 10:34:34 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:34 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:34 173-231-63-2 pmxcfs[1726]: [status] notice: cpg_send_message retried 2 times
Jun 15 10:34:45 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:45 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a73) was formed. Members
Jun 15 10:34:45 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:45 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:34:56 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:34:56 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a77) was formed. Members
Jun 15 10:34:56 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:34:56 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:02 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:02 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a7b) was formed. Members
Jun 15 10:35:02 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:02 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:13 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:13 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a7f) was formed. Members
Jun 15 10:35:13 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:13 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:13 173-231-63-2 pmxcfs[1726]: [status] notice: cpg_send_message retried 4 times
Jun 15 10:35:24 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:24 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a83) was formed. Members
Jun 15 10:35:25 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:25 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:35 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:35 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a87) was formed. Members
Jun 15 10:35:36 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:36 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:46 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:46 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a8b) was formed. Members
Jun 15 10:35:47 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:47 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jun 15 10:35:58 173-231-63-2 corosync[1814]:   [QUORUM] Sync members[4]: 2 5 6 7
Jun 15 10:35:58 173-231-63-2 corosync[1814]:   [TOTEM ] A new membership (2.13a8f) was formed. Members
Jun 15 10:35:58 173-231-63-2 corosync[1814]:   [QUORUM] Members[4]: 2 5 6 7
Jun 15 10:35:58 173-231-63-2 corosync[1814]:   [MAIN  ] Completed service synchronization, ready to provide service.

pvecm status
On node already in cluster - not showing
Code: 
Cluster information
-------------------
Name:             [REDACTED]
Config Version:   19
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sat Jun 15 10:29:01 2024
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000007
Ring ID:          2.139f7
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      4
Quorum:           3 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 [REDACTED]
0x00000005          1 [REDACTED]
0x00000006          1 [REDACTED]
0x00000007          1 [REDACTED] (local)

On joining node - Cluster name matches
Code: 
Cluster information
-------------------
Name:             [REDACTED]
Config Version:   19
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sat Jun 15 10:30:49 2024
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1.139ab
Quorate:          No

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      1
Quorum:           3 Activity blocked
Flags:           

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 [REDACTED] (local)

journalctl -b -u pveproxy -u pvedaemon -u pve-cluster -u corosync

Code: 
Jun 15 10:33:32 192-69-221-162 corosync[2182]:   [TOTEM ] A new membership (1.13a57) was formed. Members
Jun 15 10:33:32 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 30
Jun 15 10:33:33 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 40
Jun 15 10:33:34 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 50
Jun 15 10:33:35 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 60
Jun 15 10:33:36 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 70
Jun 15 10:33:37 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 80
Jun 15 10:33:38 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 90
Jun 15 10:33:39 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 100
Jun 15 10:33:39 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retried 100 times
Jun 15 10:33:39 192-69-221-162 pmxcfs[2184]: [status] crit: cpg_send_message failed: 6
Jun 15 10:33:40 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 10
Jun 15 10:33:41 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 20
Jun 15 10:33:42 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 30
Jun 15 10:33:43 192-69-221-162 corosync[2182]:   [QUORUM] Sync members[1]: 1
Jun 15 10:33:43 192-69-221-162 corosync[2182]:   [TOTEM ] A new membership (1.13a5b) was formed. Members
Jun 15 10:33:43 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 40
Jun 15 10:33:44 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 50
Jun 15 10:33:45 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 60
Jun 15 10:33:46 192-69-221-162 pmxcfs[2184]: [status] notice: cpg_send_message retry 70

Not sure if any more information is needed to help, but any help is appreciated!

Thanks!
 
Resolved

/etc/pve/corosync.conf

config version was different on one of the nodes
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back