Ceph Recovery after all monitors are lost?

[Fred Saunier]

Hello,

My 3 ceph monitors (v 15.2.14) have all crashed after their store.db have all become corrupted, to the point the only way I could see out of this was to destroy and re-create those monitors from scratch. The OSDs have been untouched since the crash, they stand in the state they were in just before data corruption (when ceph health was still OK).

I have tried rebuilding the store.db from the OSDs, using this script : https://docs.ceph.com/en/latest/rad...leshooting-mon/#mon-store-recovery-using-osds -- after some tweaking to get the paths right (to the best of my knowledge, but the paths may be off), the script executes, a new store.db gets created but when I move it the the 3 monitors, nothing happens (the OSDs do not appear, nor the pools). Rebooting the 3 hypervisors did not help.

The script above references the keyring /path/to/admin.keyring
I translated it to /etc/pve/priv/ceph.mon.keyring
Is that correct?

How can I re-instert the existing OSDs and the pools into the crush map? I have seen this thread (https://forum.proxmox.com/threads/return-existing-osd-to-crush-map-how.43800/). Does it apply to Octopus? Can I just re-add all OSDs one by one and recover my pools?

 

[Fred Saunier]

I was fortunate enough to have a backup copy of my crushmap (which I'll make sure I'll update after any modification to the OSDs). Here are the steps I took to restore my ceph environment successfully:

1. Restore crushmap

ceph osd setcrushmap -i backup-crushmap

The crush map now displays all OSDs, as before the crash.

2. Stop all monitors and managers (from the GUI)

3. Run the Monitor recovery using ODSs script referenced above

4. Delete the /var/lib/ceph/mon/ceph-<monitor_id>/store.db on each monitor

5. Copy the /root/mon-store/store.db folder (created by the script) to each monitor (/var/lib/ceph/mon/ceph-<monitor_id>/store.db

6. chown -R ceph:ceph /var/lib/ceph/mon/ceph-<monitor_id>/store.db (on each monitor)

7. start all monitors and managers (from the GUI)

8. start all OSDs (from the GUI)

 

[DemiNe0]

How would you recover from this without a backup of the crush map?

 

[GoZippy]

I somehow lost all my osd and map too - when I did pm gui update.. after reboot everything went to hell... any ideas on any of this?

ceph osd setcrushmap -i backup-crushmap

and just about any command for ceph just hangs and or times out...

Monitors are listed but no quorum
No OSDs are listed
No Managers are listed anymore

obvious issues...

need steps to flag out somehow (again cli ceph command times out)
noup
and start rebuilding node 1 and then node 2 and then let it resync the 9 nodes I have... or any other ideas?

 

[ncrubyguy]

I somehow lost all my osd and map too - when I did pm gui update.. after reboot everything went to hell...

This is my situation as well. I upgraded, and upon restart every system SSD developed hardware media errors right in the middle of the rocksdb used by the monitors.

Something is fishy

 

[DemiNe0]

When this problem happened to me back in Jan(also caused by a proxmox upgrade), the only way I was able to recover was by rebuilding each node, one by one.

 

[ncrubyguy]

When this problem happened to me back in Jan(also caused by a proxmox upgrade), the only way I was able to recover was by rebuilding each node, one by one.

I assume you had backups like a sensible person. This was a pretty new cluster (March) with just a handful of RBD VMs on it, but few weren't backed up yet. My strategy now is to somehow reassemble the VM images from the intact OSDs and then start again.

 

[GoZippy]

yeah I am trying to reduce down to just one monitor and manager and then add back but nothign seems to be helping... so much frustration over last 3 months now since upgrade... I did have some vm I would like to restore and save so only reason I am trying so hard not to lose everything.

 

[ncrubyguy]

If anyone is curious, I ended up doing this. All rbd images 100% recovered. I've since rebuilt the cluster using mirrored ZFS on hard drives for the OS volume.

 

[imoniker]

If anyone is curious, I ended up doing this. All rbd images 100% recovered. I've since rebuilt the cluster using mirrored ZFS on hard drives for the OS volume.

Hello, do you have any updates on the stability of ZFS？Ceph seems to be fragile.