Addings OSDs to Ceph - 1 vs several ?

[Fred Saunier]

Hi all,

We are going to add ODSs to our current CEPH (nautilus 14.2.10). CEPH currently holds 20 disks on 5 nodes, and we are adding 10 more OSDs.

My question is, which is better? Adding one OSD at a time, and wait for re-balance before adding the next one? Or adding all 10 OSDs all at once at wait for rebalance? Or adding all OSDs one node at a time? We are interested in maintaining performance for the users.

Am interested in your experience.

Thanks,
Fred

 

[Alwin]

The rebalance is a background task and should not interfere with client access. Ceph has to calculate the PG distribution in any case and then it is transferring the PGs to their new OSDs.

If you want to control data movement then you can set the weight to 0 when you add the new OSDs. And then increment it after you added all of them till their target weight is set. But with decent hardware you don't need to think to much about it.
https://docs.ceph.com/docs/master/rados/operations/monitoring-osd-pg/#back-filling