This morning node 3 of my 3 node cluster had a hiccup, not sure why, but after, the ceph monitor running on it would not start.
destroyed the monitor and recreated, but the newly created monitor is not 'whole'. It wont start and the ceph cluster does not see it.
Proxmox reports its there but it has no ':6789/0' entry.
I can remove the 'none participating monitor' by editing /etc/ceph/ceph.conf, removing the link in '/etc/systemd/system/ceph-mon.target.wants' and moving the '/var/lib/ceph/mon/' entry.
From my reading it looks like the monitor is still defined in the monmap in the other monitors, so I am going to attempt to 'clear' the now dead monitor entry by using the monmaptool to remove it.
The first step of this documented process, located at https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/, is to stop the monitors on the other cluster nodes.
My question and concern is, can I actually stop all the monitors without harming the running ceph cluster ?
I would appreciate any feed back of someone with experience with ceph clusters.
Thanks
destroyed the monitor and recreated, but the newly created monitor is not 'whole'. It wont start and the ceph cluster does not see it.
Proxmox reports its there but it has no ':6789/0' entry.
I can remove the 'none participating monitor' by editing /etc/ceph/ceph.conf, removing the link in '/etc/systemd/system/ceph-mon.target.wants' and moving the '/var/lib/ceph/mon/' entry.
From my reading it looks like the monitor is still defined in the monmap in the other monitors, so I am going to attempt to 'clear' the now dead monitor entry by using the monmaptool to remove it.
The first step of this documented process, located at https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/, is to stop the monitors on the other cluster nodes.
My question and concern is, can I actually stop all the monitors without harming the running ceph cluster ?
I would appreciate any feed back of someone with experience with ceph clusters.
Thanks