PVE - Remove one node from ceph cluster

[B3rt_PX]

Hello,
I have a PVE cluster with ceph installed.
My ceph cluster is so composed:
-node1: ceph manager, ceph monitor and osd
-node2: ceph manager, ceph monitor and osd
-node3: ceph manager, ceph monitor and osd
-node4: ceph manager, ceph monitor and osd

I need to remove one node(either ceph and pve) from cluster and my questions are:

1)Where can I find the procedure to remove the node ceph in the cluster? It only necessary from GUI I destroy the Monitor/Manager role and after remove the OSDs?Do I need to do anything else?


2)After executing the step 1, what is the correctly procedure to remove a PVE node? Is it just needs to run the command "pvecm delnode hp4"?(I take command here https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node )

3)Do I have to do all this from CLI o GUI?

4)Is possible do it safely, without deleting important data?

PVE node version: 7.3-3
Ceph OSD version: 16.2.7

Thanks

 

[aaron]

1)Where can I find the procedure to remove the node ceph in the cluster? It only necessary from GUI I destroy the Monitor/Manager role and after remove the OSDs?Do I need to do anything else?

You can stop and destroy/remove the MGR and MON directly. For the OSDs I would recommend that you first set them to "out", but leave them running. This gives Ceph time to move the data to the other nodes in the cluster. Once the OSDs are empty, you can stop and destroy them.

2)After executing the step 1, what is the correctly procedure to remove a PVE node? Is it just needs to run the command "pvecm delnode hp4"?(I take command here https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node )

Pretty much this. If the node will never come back to the cluster, you could also cleanup the Ceph CRUSH map and remove the host bucket. Should be something like

ceph osd crush remove <hostname>

3)Do I have to do all this from CLI o GUI?

It all can be done on the CLI, but especially the Ceph part as described in response to question 1 can be done in the GUI.

4)Is possible do it safely, without deleting important data?

If you follow the procedure regarding the OSD removal, you should always have enough redundancy. Just make sure that the remaining 3 nodes in the cluster have enough space to take up the data of the node that is removed. Also that you still have enough space afterwards. Ceph can handle a lot, but running out of space is one of the few really painful situations.

 

[B3rt_PX]

Pretty much this. If the node will never come back to the cluster, you could also cleanup the Ceph CRUSH map and remove the host bucket. Should be something like

ceph osd crush remove <hostname>

Thanks for the reply @aaron, It was all clear.

I have the latest question:
After I clean up the Ceph CRUSH map and remove the host bucket, there's no problem if I add a new node with the same hostname and using the same disks as replaced server, right?

Thanks

 

[aaron]

After I clean up the Ceph CRUSH map and remove the host bucket, there's no problem if I add a new node with the same hostname and using the same disks as replaced server, right?

Nope, that should work fine. But if you plan to add another server to the cluster with the same hostname very soon, you could just leave the host bucket. It might show in the OSD overview, but should not contain any OSDs (as you destroyed them earlier).

 

[B3rt_PX]

Hi @aaron,
thanks for answering my question.

If I add new cluster node (either PVE and CEPH) and I want to keep the same CEPH version (16.2.7), it's possible even if the latest version is 16.2.9?
In brief, I need to install the 16.2.7 ceph version and not 16.2.9.

Thanks

 

[aaron]

If I add new cluster node (either PVE and CEPH) and I want to keep the same CEPH version (16.2.7), it's possible even if the latest version is 16.2.9?
In brief, I need to install the 16.2.7 ceph version and not 16.2.9.

Only if you go through the manual labor of checking which packages would be installed (including dependencies) and manually specifying the exact version you want installed.
Otherwise, the recommened procedure would be to update the cluster before you add the new node.