[SOLVED] changed CEPH public network ... failed

[kifeo]

Hello,

In order to change the ceph public network, I followed this : https://ceph.readthedocs.io/en/latest/rados/operations/add-or-rm-mons/
Of course, the messy way, as I had also to change the network in ceph.conf.

it failed when I wanted to restart the monitors.

I wanted to recreate the monitor with the pveceph createmon.
I have : 'Could not connect to ceph cluster despite configured monitors'.

I have no other information and nothing is created in /var/lib/ceph/mon/

Could someone please help ?

 

[kifeo]

ok, I created the monitor with the manual procedure on two nodes in order to get the quorum.

the proxmox interface could communicate with the ceph cluster again.
I destroyed all the manager with the GUI and created them again.
As well with the Metadata servers.

However .... all the OSD are gone and I have the error when I launch the command 'systemctl restart ceph-osd.target' :

"monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]"
"failed to fetch mon config (--no-mon-config to skip)"

not really verbose to me.

Thanks !

 

[kifeo]

I also wanted to mention that creating the monitor the manual way is set as root:root user for permissions of content in /var/lib/ceph/mon/ceph-proxmox5/store.db/, whereas it is ceph:ceph when created with the GUI.

 

[kifeo]

my crushmap is also empty from osd and rules .... this begins to smell bad

The osd service that still run on non-rebooted services report healthcheck failed on the old IPs.

 

[kifeo]

"failed to fetch mon config (--no-mon-config to skip)" is due to the mon keyring in /var/lib/ceph/osd/ceph-5/keyring not added in the ceph auth list.
I added it with "ceph auth add osd.5 osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-5/keyring"

However, it is not giving the crush map

 

[kifeo]

I could not handle it myself, so I reverted to the previous backup of /var/lib/ceph/mon/ceph-proxmox and reinjected the monmap.

Regreated the mgr and mds and here are we back online.

 

[GoZippy]

can you give me steps and commands you use to revert to backup and reinject monmap so I can get my managers and mds back and hopefully rebuild data from the ODS locations?

I have similar issue in another thread with all my details.... If you have any help I would appreciate it. How did you fix?

https://forum.proxmox.com/threads/ceph-down-if-one-node-is-down.51033/