Ceph - can't remove monitor with "unknown" status

ozdjh

Well-Known Member
Oct 8, 2019
114
26
48
Hi

I added a new node to our cluster. This node will run ceph but not run a monitor or manager or have any OSDs (it's just a 'client' so we can export ceph volumes to local storage). When installing ceph and adding it to the cluster it came up with a monitor. I stopped and destroyed the monitor but it's still showing in the GUI with a status of "unknown" and an address of "unknown".

  1. Trying to destory it through the UI fails with 'no such monitor'.
  2. 'ceph mon rm' also says the monitor does not exist. As does 'pvecph mon destroy'
  3. 'ceph mon dump' does not show the new node, just the main 4 nodes
  4. /etc/pve/ceph.conf does not list the IP of the new node in the mon_host config
  5. There was a remnant file in /etc/systemd/system/ceph-mon.target.wants/ that's been removed
  6. There are no ceph processes running on the new node and it's been rebooted several times
  7. 'ceph -s' reports HEALTH_OK and lists the normal 4 monitors, not the new / dead one.

Where is the UI getting the list of monitors? I assumed it did the equivalent of a 'ceph mon dump' but that doesn't appear to be the case. I can't find any way to remove this "unknown" monitor. Any pointers would be appreciated.


Thanks
David
 
Hi,

maybe the GUI has cached it.
Did you clear the browser cache?
 
Did you remove the mon dir in /var/lib/ceph?
 
  • Like
Reactions: mcgarrah
Bingo ! Thanks wolfgang. That was it. The mgr directory was empty but the mon directory still had an entry on the new node. I removed that and the UI is now happy.

Thanks

David
...
 
Bingo ! Thanks wolfgang. That was it. The mgr directory was empty but the mon directory still had an entry on the new node. I removed that and the UI is now happy.

Thanks

David
...
Hello. I got the same problem. I see that all monitor is "unknow" now. May i know more detail how to solve the problem?

Parker
 
Hello. I have used below command can remove the mon in the ui but it seem i can't to add the host back to mon. I got the message
"

monitor 'mon.cccs01' already exists (500)
"

Any suggestion. Thank you.
 
Did you remove the mon dir in /var/lib/ceph?
That solved my problem as well... it was driving me insane.

Just make sure to keep the top level directory /var/lib/ceph/mon/ still there and owned by ceph:ceph or you get another error. Only delete the contents of the directory.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!