Cannot recreate mon and have it show up in interface and "ceph -s"

fxandrei

Renowned Member
Jan 10, 2013
146
12
83
So i dont know what exactly happened but today i saw that one of the monitors on one of my notes was stopped.
I tryed to start it but nothing.
I rebooted the node, and it still did not start.

The problem was that the ceph-mon@xxx.service was running, just that the cluster was saying it was down.
So pveceph destroy did not work (it sayd it did not find the id).

So i ended up doing this:
- disable the ceph-mon service
- manually remove the ip of the mon host in /etc/pve/ceph.conf along with [mon.xxxx] entry (it then synced to the other nodes)
- run: rm -rf /var/lib/ceph/mon/ceph-xxxxx
- rebooted the node

Then if i try to create the mon again i can see that the service is created, the ceph.conf is changed, and the ceph-mon@xxx.service is running, but it does not show up in ceph-s

Did anyone had this kind of problems ?
What can i do\check ?
 
Well this was kind of weird. So i ended up trying to reboot the other nodes, and when i tried that with the first one, the mon did not start.
When i checked the logs i got an error like this:
rocksdb: IO error: While opening a file for sequentially reading: /var/lib/ceph/mon/ceph-xxx/store.db/CURRENT: Permission denied

So i checked that directory and saw that some files including the CURRENT file was owned by root. I dont know how this happened.
I then changed the owner of all of them and rebooted again. This time the mon started.

I then tried to recreate the mon on the initial node, and now it worked.

It would seem that the mon on the first node was stuck somehow (could not write maybe? ) and when i added the mon on the other node, it created the service and everything, it just could not sync the db across the mons, i think.

Has anyone encountered this ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!