[SOLVED] How to destroy a CEPH OSD on a node already removed from cluster?

[proxwolfe]

Hi,

So I removed a node from my PVE cluster. That worked well and as expected.

But when I tried to then destroy the monitor and the OSD that used to be running on this node, I got error messages that indicate that CEPH can't find the node anymore (which, of course, makes sense).

So the next time, I will probably do it the other way around, first destroy OSD and monitor and then remove the node from the PVE cluster. But for now, how do I get rid of the monitor and the OSD that CEPH is still looking for but won't let me destroy/remove?

Thanks!

 

[aaron]

Since the node is gone, no manual cleanup of the actual OSD service is needed. You just need to clean up the Ceph CRUSH map side of things.
Check out the Ceph documentation on how to manually remove an OSD.
The TL;DR should be:

ceph osd purge {id} --yes-i-really-mean-it

You will also see a bucket in the CRUSH Map for the node itself. For example in the GUI under Node -> Ceph -> Configuration on the right side.

Once all the OSDs that used to be in the node are gone, you can remove the node bucket with (Ceph documentation)

ceph osd crush remove {bucket-name}

After that, the Crush map should have no mention of the OSDs or node that are no more.

 

[proxwolfe]

Worked (even without the "--yes-i-really-mean-it" - I thought I'd do a dry run without it first, but the OSD was removed all the same).

Thanks a lot!!!

 

[aaron]

Good to hear . I went ahead and marked the thread as solved. You could do so as well by editing the first post and selecting the prefix from the drop down menu next to the title.

 

[proxwolfe]

I'am afraid I need to come back to this once more:

While the host has been purged from the "bucket list" , it is still living on in the GUI (main CEPH view, "services" block under "Monitors" and in the CEPH OSD view in the list of Monitors but nowhere else, as far as I can tell).

Is there a way to remove it from there, too?

Thanks!

 

[proxwolfe]

The problem seems to have to do with the fact that while the old node ("bucket") is gone from the Crush Map, it still shows up in Configuration with two entries: mon.oldnode and mds.oldnode.

Is it possible to remove those entries either with some command or directly from the config file?

Thanks!

 

[proxwolfe]

okay, found it:

ceph mon remove <oldnode>

does the trick

 

[Gilberto Ferreira]

Before remove the node is not a good practice do an out and than stop, waiting for CEPH rebalance and only than stop, remove and purge the osd?