Replacing a Node in a Ceph Cluster

[bcops]

We have a cluster of 3 nodes running Proxmox in a HA Ceph cluster with a 10GbE mesh network for Ceph. We are wanting to completely replace one of the nodes in the cluster, and was wondering what the best way of going about doing that would be? I tried looking through the documentation about this, but was unable to find anything that seemed to address it specifically. Any assistance would be greatly appreciated.

 

[ph0x]

You can just replace the node. Maybe set the OSD flags to noout, nobackfill before but Ceph will handle the rest.
There's enough documentation on how to detach and destroy OSDs, but it doesn't matter if you do this while they're in or out. Same goes for the node itself.

 

[lDemoNl]

We have a cluster of 3 nodes running Proxmox in a HA Ceph cluster with a 10GbE mesh network for Ceph. We are wanting to completely replace one of the nodes in the cluster, and was wondering what the best way of going about doing that would be? I tried looking through the documentation about this, but was unable to find anything that seemed to address it specifically. Any assistance would be greatly appreciated.

Dont forget about quorum, 3 nodes in cluster. This way you can create 1 virtual node, after adding new node delete VM

 

[ph0x]

That is for the faint-hearted. Replacement of a node should take minutes. You must have more than bad luck if another node fails in exact this time ...

 

[bcops]

You can just replace the node. Maybe set the OSD flags to noout, nobackfill before but Ceph will handle the rest.
There's enough documentation on how to detach and destroy OSDs, but it doesn't matter if you do this while they're in or out. Same goes for the node itself.

Thanks for your reply. To clarify some things: is your suggestion to basically down the node we want to replace, force it to rejoin the cluster using the Recovery documentation, and then recreate the OSDs and then wait for the Ceph pool to reach a recovered state? Just wanting to make sure we're taking everything into account to try and avoid any major pitfalls or so.

 

[ph0x]

That's what I would do, yes. If you want to reuse the OSDs there might even be a way to restore them without having the cluster go through a backfill. I didn't dig into the necessary steps so far, but that should be possible. If you deploy a completely new node I'd just create the new OSDs, destroy the old ones that were removed and the enable the backfilling again.