Node fell out of cluster, can't re-join.

alexgo

New Member
Aug 17, 2013
2
0
1
We've got a three-node proxmox cluster running 2.3. Recently, one of the nodes seems to have fallen out of the cluster (/etc/pve has gone read-only, is not getting changes from the other cluster nodes). Here's the "pvecm status" output from a good node:

Code:
Version: 6.2.0
Config Version: 3
Cluster Name: 458cluster
Cluster Id: 60724
Cluster Member: Yes
Cluster Generation: 536
Membership state: Cluster-Member
Nodes: 2
Expected votes: 3
Total votes: 2
Node votes: 1
Quorum: 2  
Active subsystems: 5
Flags: 
Ports Bound: 0  
Node name: vmhost1
Node ID: 3
Multicast addresses: 239.192.237.34 
Node addresses: 172.20.1.2

And here's the failed node:

Code:
Version: 6.2.0
Config Version: 3
Cluster Name: 458cluster
Cluster Id: 60724
Cluster Member: Yes
Cluster Generation: 532
Membership state: Cluster-Member
Nodes: 1
Expected votes: 3
Total votes: 1
Node votes: 1
Quorum: 2 Activity blocked
Active subsystems: 1
Flags: 
Ports Bound: 0  
Node name: vmhost2
Node ID: 2
Multicast addresses: 239.192.237.34 
Node addresses: 172.20.1.3

How can I make the orphaned node rejoin the other two cluster members?

Thanks!

EDIT: Also, I've run the following to try and get vmhost2 to re-join the cluster:
Code:
/etc/init.d/cman stop ; /etc/init.d/pve-cluster stop
/etc/init.d/pve-cluster start ; /etc/init.d/cman start
 
Last edited:
We've got a three-node proxmox cluster running 2.3. Recently, one of the nodes seems to have fallen out of the cluster (/etc/pve has gone read-only, is not getting changes from the other cluster nodes). Here's the "pvecm status" output from a good node:

Code:
Version: 6.2.0
Config Version: 3
Cluster Name: 458cluster
Cluster Id: 60724
Cluster Member: Yes
Cluster Generation: 536
Membership state: Cluster-Member
Nodes: 2
Expected votes: 3
Total votes: 2
Node votes: 1
Quorum: 2  
Active subsystems: 5
Flags: 
Ports Bound: 0  
Node name: vmhost1
Node ID: 3
Multicast addresses: 239.192.237.34 
Node addresses: 172.20.1.2

And here's the failed node:

Code:
Version: 6.2.0
Config Version: 3
Cluster Name: 458cluster
Cluster Id: 60724
Cluster Member: Yes
Cluster Generation: 532
Membership state: Cluster-Member
Nodes: 1
Expected votes: 3
Total votes: 1
Node votes: 1
Quorum: 2 Activity blocked
Active subsystems: 1
Flags: 
Ports Bound: 0  
Node name: vmhost2
Node ID: 2
Multicast addresses: 239.192.237.34 
Node addresses: 172.20.1.3

How can I make the orphaned node rejoin the other two cluster members?

Thanks!

EDIT: Also, I've run the following to try and get vmhost2 to re-join the cluster:
Code:
/etc/init.d/cman stop ; /etc/init.d/pve-cluster stop
/etc/init.d/pve-cluster start ; /etc/init.d/cman start
Hi,
have you also tried an
Code:
service pvedaemon restart
Do ssmping work? http://pve.proxmox.com/wiki/Multicast_notes

Udo
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!