Ceph monitor on one node (out of three) won't start.

[Batpenguin]

This below is from the syslog. All of my OSDs are good. See attached picture. All I have is one monitor on my 2nd server that won't start. Can I just destroy that monitor and recreate it?

Feb 04 23:39:53 pve2 systemd[1]: Started Ceph cluster monitor daemon.
Feb 04 23:39:53 pve2 ceph-mon[2834]: 2020-02-04 23:39:53.397 7f2d4ad30440 -1 rocksdb: Corruption: Can't access /227675.sst: IO error: No such file or directorywhile stat a file for size: /var/lib/ceph/mon/ceph-pve2/store.db/227675.sst: No such file or directory
Feb 04 23:39:53 pve2 ceph-mon[2834]: 2020-02-04 23:39:53.397 7f2d4ad30440 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-pve2': (22) Invalid argument
Feb 04 23:39:53 pve2 systemd[1]: ceph-mon@pve2.service: Main process exited, code=exited, status=1/FAILURE
Feb 04 23:39:53 pve2 systemd[1]: ceph-mon@pve2.service: Failed with result 'exit-code'.
Feb 04 23:40:03 pve2 systemd[1]: ceph-mon@pve2.service: Service RestartSec=10s expired, scheduling restart.
Feb 04 23:40:03 pve2 systemd[1]: ceph-mon@pve2.service: Scheduled restart job, restart counter is at 6.
Feb 04 23:40:03 pve2 systemd[1]: Stopped Ceph cluster monitor daemon.
Feb 04 23:40:03 pve2 systemd[1]: ceph-mon@pve2.service: Start request repeated too quickly.
Feb 04 23:40:03 pve2 systemd[1]: ceph-mon@pve2.service: Failed with result 'exit-code'.
Feb 04 23:40:03 pve2 systemd[1]: Failed to start Ceph cluster monitor daemon.

 

[Alwin]

Feb 04 23:39:53 pve2 ceph-mon[2834]: 2020-02-04 23:39:53.397 7f2d4ad30440 -1 rocksdb: Corruption: Can't access /227675.sst: IO error: No such file or directorywhile stat a file for size: /var/lib/ceph/mon/ceph-pve2/store.db/227675.sst: No such file or directory Feb 04 23:39:53 pve2 ceph-mon[2834]: 2020-02-04 23:39:53.397 7f2d4ad30440 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-pve2': (22) Invalid argument

The MON DB seems to be corrupt. The easiest path is to destroy and then create the MON again.

 

[Batpenguin]

The MON DB seems to be corrupt. The easiest path is to destroy and then create the MON again.

In doing so am I at risk of any data loss? I have backups of everything

 

[Alwin]

In doing so am I at risk of any data loss?

Usually not. I think you may have some Ceph architecture reading to do.
https://docs.ceph.com/docs/nautilus/architecture/