Ceph Ghost Monitor from removed node

[apauna]

I removed node 4 from my ProxMox Cluster due to failure of node. Now I have a ghost Ceph monitor stuck in the cluster. I have tried may ways to clear this but it is still stuck in my cluster. Running ciph mon dump show mon.pve4 and that is no longer in the cluster. Attempted destroy and it is not able to remove. How do I remove a ghost ceph monitor?

ceph mon dump
epoch 17
fsid abc349bc-90dd-406b-9618-9cf27221efa6
last_changed 2024-01-21T16:41:21.211651-0600
created 2022-09-20T17:35:57.964052-0500
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:192.168.6.4:3300/0,v1:192.168.6.4:6789/0] mon.pve4
1: [v2:192.168.6.6:3300/0,v1:192.168.6.6:6789/0] mon.pve6
2: [v2:192.168.6.7:3300/0,v1:192.168.6.7:6789/0] mon.pve7
3: [v2:192.168.6.1:3300/0,v1:192.168.6.1:6789/0] mon.pve01
4: [v2:192.168.6.2:3300/0,v1:192.168.6.2:6789/0] mon.pve2
dumped monmap epoch 17

 

[Alwin Antreich]

You can remove the MON by issuing the ceph mon rm pve4 on the CLI. The UI will likely not handle this specific case.

Please be aware that an even number of MONs is not recommended, as it has higher likelihood of running into a split brain (loss of quorum).

 

[Maximiliano]

Hello,

As already said, you can destroy this monitor with

ceph mon remove pve4

And you can verify if it was removed from the monmap with

ceph mon dump

If it is still there you might need to restart Ceph's monitor services.

 

[Azunai333]

lease be aware that an uneven number of MONs is not recommended, as it has higher likelihood of running into a split brain (loss of quorum).

What do you mean by that?
If you have 4, you can lose 1 (3/4).
If you have 3, you can lose 1 (2/3).
You get a higher risk of failure with an even number of MONs. Shouldn't it be recommended to use an odd number of MONs?

 

[gfngfn256]

What do you mean by that?

I believe it is a typo & @Alwin Antreich meant "even number" ; which is where your Ceph MON's will be after removing PVE4

As per Ceph Docs

 

[Alwin Antreich]

I believe it is a typo & @Alwin Antreich meant "even number" ; which is where your Ceph MON's will be after removing PVE4

As per Ceph Docs

Thanks for noticing. I've updated my post.

 

[apauna]

Thanks for noticing. I've updated my post.

Thanks for the response and yes you are correct I had 4 running and that is not ideal. I have moved it down to three now since I have been able to remove the offending pve4.

That is very strange that I was making this harder then it needed to be. I was sure I tried ceph mon remove Anyway with the command correct ceph mon remove pve4 it removed it from Ceph as needed, Thank you for the help!

I still cannot believe it was that simple