OSD replacement and adding minimizing rebalances

[lucaferr]

Hello! I have a 10 nodes hyperconverged cluster. Each node has 4 OSDs (total 40 OSDs). I have 2 different questions:

QUESTION 1 - OSD REPLACEMENT (with identical SSD)
Since I need to replace an SSD (one OSD has crashed 3 times in latest months so I prefer to replace it with a brand new one) what's the best way to do it without causing rebalancing to other nodes? Since I'll replace it with an identical one I'd like a way to stop every rebalance and just backfill the new OSD once it's been replaced, with the same weight of the old one. Is the following procedure enough or do you suggest something different?

1. set "norebalance" and "nobackfill"
2. stop the OSD
3. mark the OSD out
4. destroy the OSD
5. set "noout"
6. turn off the node, unplug the old OSD, insert the new (identical) OSD, turn on the node
7. unset "noout"
8. create the new OSD
9. unset "norebalance" and "nobackfill"

QUESTION 2 - ADD MULTIPLE OSDs TO THE CLUSTER
I need to add a new node, with 4 news OSDs (2 TB each, all SSDs). Should I create one OSD, wait for rebalance to end, create the second, wait for the rebalance, etc...or create the 4 new OSDs all together? It's a production cluster so I'd like to avoid an excessive rebalance load.

Thank you very much!

 

[mira]

Your steps sound good, maybe add `norecover` as well.

When adding, add one at a time to minimize the impact on the cluster. You can also control how fast rebalancing and recovery are running [0].


[0] https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/