Ceph problem after removing node.

[300cpilot]

I have a 4 node cluster and removed one today. Ceph took 4+ hours to sync up but now shows a health warning and still shows the node that was removed as a monitor.
How do I get this cleaned up? Everything is running ok, so far.

 

[Moayad]

Hi,

Does the monitor for Node-D still exists?

 

[300cpilot]

I could add the node back in after a reload, if I really had to. But the node was deleted out of the cluster.

 

[aaron]

Did you stop and destroy the Ceph services (MON, OSDs) before you shut it down?

Because this looks like it didn't happen for the MON.

It can be cleaned up manually. You can try removing it as described in the Ceph docs: https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#removing-a-monitor-manual (Removing a Monitor (Manual)) or if that doesn't help, the following section to manually remove it from the monmap.

 

[300cpilot]

No I forgot about it before I ran the command. Thanks I will go through the steps.

Now I am unable to destroy the OSD disks via the gui.

This is what I did to get the monitor portion cleaned up.

1. First, make sure that you have a quorum of monitors in your Ceph cluster. This means that you should have at least two monitors remaining after you remove the one you want to remove.
2. SSH into the Proxmox node where the monitor you want to remove is running.
3. Run the following command to determine the ID of the monitor:

ceph mon stat

1. The output will show you the ID of each monitor in the cluster, as well as its hostname and IP address. Make note of the ID of the monitor you want to remove.
2. Run the following command to remove the monitor:

ceph mon remove <monitor-id>

1. Replace <monitor-id> with the ID of the monitor you want to remove. In my case Node-D.
2. The monitor should now be removed from the cluster. You may need to restart the Ceph daemon on the Proxmox node where the monitor was running for the changes to take effect.
3. Finally, verify that the monitor has been removed by running the following command:

ceph mon stat

This should show you a list of the remaining monitors in the cluster, and the monitor you removed should not be included.

 

[300cpilot]

To remove an OSD:
Note there is a waiting time to resync if you have not already done it. Let it finish! (Steps 7 & 8)
Mine was already done.

1. First, make sure that you have at least one other OSD in the cluster, as well as a quorum of monitors.
2. SSH into the Proxmox node where the OSD you want to remove is running.
3. Run the following command to determine the ID of the OSD:

ceph osd tree

1. The output will show you a tree-like structure, with each OSD represented by a number. Make note of the ID of the OSD you want to remove.
2. Run the following command to mark the OSD for removal:

ceph osd out <osd-id>

1. Replace <osd-id> with the ID of the OSD you want to remove. In my case 0-5.
2. The OSD will now be marked for removal, but it will not be removed from the cluster until the data it contains has been rebalanced to other OSDs.
3. To check the status of the rebalancing process, you can run the following command:

ceph -w

1. This will show you the status of the cluster, including the progress of the rebalancing process.
2. Once the rebalancing process is complete, you can run the following command to remove the OSD from the cluster:

ceph osd crush remove osd.<osd-id>
ceph osd rm <osd-id>

1. Replace <osd-id> with the ID of the OSD you want to remove.
2. The OSD should now be removed from the cluster. You may need to restart the Ceph daemon on the Proxmox node where the OSD was running for the changes to take effect.
3. Finally, verify that the OSD has been removed by running the following command:

ceph osd tree

This should show you a list of the remaining OSDs in the cluster, and the OSD you removed should not be included.

Now to get rid of the Host-D...

 

[300cpilot]

This last step appears to be a time issue. It is now rebalancing again. Note that in my case all of my vm's and ct's are off line, even if the GUI has them running..

I did have to remove the server from /etc/ceph/ceph.conf. (Make a backup first! cp /etc/ceph/ceph.conf /etc/ceph/ceph.conf.bak)

I will update when the resync is finished in a couple of hours.