Cannot recreate mon and have it show up in interface and "ceph -s"

[fxandrei]

So i dont know what exactly happened but today i saw that one of the monitors on one of my notes was stopped.
I tryed to start it but nothing.
I rebooted the node, and it still did not start.

The problem was that the ceph-mon@xxx.service was running, just that the cluster was saying it was down.
So pveceph destroy did not work (it sayd it did not find the id).

So i ended up doing this:
- disable the ceph-mon service
- manually remove the ip of the mon host in /etc/pve/ceph.conf along with [mon.xxxx] entry (it then synced to the other nodes)
- run: rm -rf /var/lib/ceph/mon/ceph-xxxxx
- rebooted the node

Then if i try to create the mon again i can see that the service is created, the ceph.conf is changed, and the ceph-mon@xxx.service is running, but it does not show up in ceph-s

Did anyone had this kind of problems ?
What can i do\check ?

 

[fxandrei]

Well this was kind of weird. So i ended up trying to reboot the other nodes, and when i tried that with the first one, the mon did not start.
When i checked the logs i got an error like this:
rocksdb: IO error: While opening a file for sequentially reading: /var/lib/ceph/mon/ceph-xxx/store.db/CURRENT: Permission denied

So i checked that directory and saw that some files including the CURRENT file was owned by root. I dont know how this happened.
I then changed the owner of all of them and rebooted again. This time the mon started.

I then tried to recreate the mon on the initial node, and now it worked.

It would seem that the mon on the first node was stuck somehow (could not write maybe? ) and when i added the mon on the other node, it created the service and everything, it just could not sync the db across the mons, i think.

Has anyone encountered this ?