is it possible to recover ceph data

[imoniker]

I'm testing PVE ceph on a 3 node cluster.
Today, after several host reboots, ceph crashed with the following error:
rocksdb: Corruption: Corruption: IO error: No such file or directoryWhile open a file for random read: /var/lib/ceph/mon/ceph-pve3n1/store.db/001272.ldb: No such file or directory
I deleted mon config in ceph.cfg, and removed directory in /var/lib/ceph/mon/ for all 3 nodes. When I tried to run pveceph create mon, it returned:
unable to get monitor info from DNS SRV with service name: ceph-mon

Is there anyway to recover data on ceph?
Other ceph services, such as OSD seemed OK.

 

[gurubert]

When you removed the MON database you basically wiped your Ceph cluster.
You can try to recover the MON database from an OSD: https://docs.ceph.com/en/quincy/rad...leshooting-mon/#mon-store-recovery-using-osds

Did you only have one MON in the cluster? The recommendation is to run at least three.

 

[imoniker]

Thanks for the reply. I had 3 monitors, all of them went down. It's a testing cluster so it doesn't matter.
I searched on this forum and I also found this:
How to recover a ceph cluster if all monitors are down:
https://access.redhat.com/documenta...-ceph-monitor-for-bare-metal-deployments_diag
I'll deliberately delete the database and test a recovery in the following weeks.

But it's really annoying since I just boot and reboot the server, the ceph mon db file is lost.

I guess I need to do some crash testing (blackout the cluster, blackout servers, reboot server randomly and frequently, delete mon db file deliberately, write until ceph is full, replace nodes, unplug hard driver...) before I could really put ceph in production. It seems there is no other better choices(raid or ZFS)...

Have you done this kind of crash test before? If you could summarize some common problems and solutions it would be very great. Losing data is really annoying.

 

[gurubert]

I just boot and reboot the server, the ceph mon db file is lost.

This is definitely not normal behaviour. A node reboot does not corrupt data.

 

[imoniker]

I rebooted all nodes in less than 1 minutes frequently in my test environment.
I wanted to see if something unnormal could crash the cluster. ^^
It happened once in 30 times I guess.

 

[gurubert]

If you really want to you can corrupt a Ceph cluster but under normal circumstances this does not happen.

There are three MONs running because the probability of all of them rebooting / crashing at the same time is very very low.

 

[imoniker]

That's true. I'll continue my crash testing.
If there is anything I'll let you know.
I would rather a crash and fix happened in the testing environment.
I also need to have some experience to fix a crash if it really happens...