Ceph Ghost Monitor from removed node

A

apauna

New Member
Feb 3, 2024
9
1
3
I removed node 4 from my ProxMox Cluster due to failure of node. Now I have a ghost Ceph monitor stuck in the cluster. I have tried may ways to clear this but it is still stuck in my cluster. Running ciph mon dump show mon.pve4 and that is no longer in the cluster. Attempted destroy and it is not able to remove. How do I remove a ghost ceph monitor?

Code: 
ceph mon dump
epoch 17
fsid abc349bc-90dd-406b-9618-9cf27221efa6
last_changed 2024-01-21T16:41:21.211651-0600
created 2022-09-20T17:35:57.964052-0500
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:192.168.6.4:3300/0,v1:192.168.6.4:6789/0] mon.pve4
1: [v2:192.168.6.6:3300/0,v1:192.168.6.6:6789/0] mon.pve6
2: [v2:192.168.6.7:3300/0,v1:192.168.6.7:6789/0] mon.pve7
3: [v2:192.168.6.1:3300/0,v1:192.168.6.1:6789/0] mon.pve01
4: [v2:192.168.6.2:3300/0,v1:192.168.6.2:6789/0] mon.pve2
dumped monmap epoch 17

1711764143904.png
 
You can remove the MON by issuing the ceph mon rm pve4 on the CLI. The UI will likely not handle this specific case.

Please be aware that an even number of MONs is not recommended, as it has higher likelihood of running into a split brain (loss of quorum).
 
Last edited:
Hello,

As already said, you can destroy this monitor with

Code: 
ceph mon remove pve4

And you can verify if it was removed from the monmap with

Code: 
ceph mon dump

If it is still there you might need to restart Ceph's monitor services.
 
Alwin Antreich said:
lease be aware that an uneven number of MONs is not recommended, as it has higher likelihood of running into a split brain (loss of quorum).
Click to expand...
What do you mean by that?
If you have 4, you can lose 1 (3/4).
If you have 3, you can lose 1 (2/3).
You get a higher risk of failure with an even number of MONs. Shouldn't it be recommended to use an odd number of MONs?
 
Alwin Antreich said:
Thanks for noticing. I've updated my post.
Click to expand...
Thanks for the response and yes you are correct I had 4 running and that is not ideal. I have moved it down to three now since I have been able to remove the offending pve4.

That is very strange that I was making this harder then it needed to be. I was sure I tried ceph mon remove Anyway with the command correct ceph mon remove pve4 it removed it from Ceph as needed, Thank you for the help!

I still cannot believe it was that simple :D
 
  • Like
Reactions: Alwin Antreich
You must log in or register to reply here.
Top Bottom