replace node in cluster

[ildar94]

Hi all!
I have a pve cluster and a cef storage of 3 nodes. On one node, both OSDs are in the down and out status.
I need to replace this node with another node.
I added a new node to the proxmox cluster, installed cef.
Tell me what are my next steps?
Do I need to delete the OSD on the node I want to remove from the cluster?
Or do I first need to create an OSD on a new node and add them to the pool?

 

[Lukas Moravek]

Hi @ildar94

- First, add the new OSDs on your new node to the cluster
- Wait for the cluster to rebalance
- Only after rebalancing is complete, remove the failed OSD
- Remove Monitor / Managers from failed node
- Clean up the old node:
-- If you want to completely remove the node from the Ceph cluster, remove it from the CRUSH map: ceph osd crush rm {node-name}

Additional docs:

https://dzone.com/articles/how-to-remove-node-from-proxmox-cluster#:~:text=To remove a node from,and clean up Ceph components.&text=If you are learning Proxmox,is build a Proxmox cluster.

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

 

[ildar94]

Hi @ildar94

- First, add the new OSDs on your new node to the cluster
- Wait for the cluster to rebalance
- Only after rebalancing is complete, remove the failed OSD
- Remove Monitor / Managers from failed node
- Clean up the old node:
-- If you want to completely remove the node from the Ceph cluster, remove it from the CRUSH map: ceph osd crush rm {node-name}

Additional docs:

https://dzone.com/articles/how-to-remove-node-from-proxmox-cluster#:~:text=To remove a node from,and clean up Ceph components.&text=If you are learning Proxmox,is build a Proxmox cluster.

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

Hi Lukas,
Thank you a lot

 

[ildar94]

I just created two OSDs on a new node. The new OSDs show up as ghost OSDs. Maybe I also need to add the monitor, manager and meta services to the new node to start the cluster rebalance process?

 

[Lukas Moravek]

Can you show me the OSDs "tab" ? Did you make the new ones UP / IN ?

 

[ildar94]

Can you show me the OSDs "tab" ? Did you make the new ones UP / IN ?

No, I dont see new osd on osd tab

 

[Lukas Moravek]

Hi @ildar94

to fully understand:

Originally you had srv-pve-05, srv-pve-04, srv-pve-02. srv-pve-05 failed (osd down / out). New node is srv-pve-06, which you added in to cluster and also to CEPH cluster, but srv-pve-06 is not visible in OSDs ?

Thanks

L.

 

[Lukas Moravek]

Hello @ildar94

for my curiosity simulated your issue with to reproduce steps.

Original cluster: proxmox1A, proxmox1B, proxmox1C
1) Simulated failure on proxmox1A - detached disks with CEPH OSDs
2) Created new node (proxmox1d) and add in to the cluster + install CECP
3) Add new OSDs on (proxmox1d)

Mo issues or data los observed





How is now your situation ?

L.

 

[ildar94]

Hello @ildar94

for my curiosity simulated your issue with to reproduce steps.

Original cluster: proxmox1A, proxmox1B, proxmox1C
1) Simulated failure on proxmox1A - detached disks with CEPH OSDs
2) Created new node (proxmox1d) and add in to the cluster + install CECP
3) Add new OSDs on (proxmox1d)

Mo issues or data los observed

View attachment 82913

View attachment 82915

How is now your situation ?

L.

Hi Lukas,
Thank you for you help.
I just had to reboot the new node. After that I saw new OSDs and the rebalancing process started.

 

[Lukas Moravek]

Hi @ildar94

good to hear.

If you want, remove the CEPH and Cluster node. Remove old OSDs, Monitor / Manager ... from failed node first.

ceph osd crush remove hostname

pvecm delnode hostname

https://forum.proxmox.com/threads/remove-node-from-cluster.98752/
https://forum.proxmox.com/threads/pve-remove-one-node-from-ceph-cluster.122456/