[SOLVED] Adding nodes while one member offline

[bensode]

I had a 5 node cluster with Ceph setup and running. Node2 was shutdown for maintenance and while that node was down I added two new nodes to expand to 7 nodes. Now that Node2 has been brought back online it does not appear in the cluster and is acting like a stand-alone PVE server. The other nodes in the cluster still show that node2 as down.



To make matters worse, it was also functioning as a Monitor for Ceph and is not joining the Ceph cluster either and reporting all OSDs down and not healthy. I used the GUI to remove the monitor while logged into another node and it looks like that worked as I'm only seeing two monitors and Ceph reports healthy. Is there any way to rejoin this node correctly without having to rebuild the entire cluster?

 

[bensode]

Actually if possible how can I completely remove this node from the PVE and Ceph cluster? The proc needs to be replaced and that's going to take at least a week. I'm thinking I will need to rebuild the cluster entirely which would suck and kill off the remainder of my day.

 

[dcsapak]

maybe this helps: https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

 

[bensode]

I had to rebuild the cluster as something got out of sync fortunately there was nothing of value yet installed. We will make note to make sure not to make cluster-wide changes with nodes offline without taking precautions.

 

[elmacus]

I did repair this error by just manually syncing the
/etc/pve/corosync.conf on my at the time offline server.
After that it was in sync and working normal.

You dont need to reinstall for this simple error.

 

[bensode]

I did repair this error by just manually syncing the
/etc/pve/corosync.conf on my at the time offline server.
After that it was in sync and working normal.

You dont need to reinstall for this simple error.

I am still learning That i good to know and I will remember thank you.