Ceph OSDs marked out, but still rebalance when I remove them.

[aarcane]

Okay, so reading the manual for Ceph tells me to mark an OSD as "Out" and then allow the pool to rebalance the data away from it before removing it, and then removing it should not require any rebalancing. Okay, so I did this. I marked an OSD as OUT. I added two OSDs to replace it. I let the rebalance run. For two days, I let it run, until everything was 100% active+clean, the new OSDs were up and in, the old OSD was up and OUT. I deleted the old OSD using the GUI. Immediately another day long OSD rebalance occured. What did I do wrong, or is this something wrong with ceph or is my understanding wrong, or what?

 

[wigor]

Hey aarcane,

iirc you better set the crush weight of the "leaving" osd to zero. So the weight of the host is also altered.
Otherwise the second rebalancing occured due to altered host weight after destroying osd. Setting osd out on it´s own does not alter the host weight (iirc).

 

[aarcane]

Hey aarcane,

iirc you better set the crush weight of the "leaving" osd to zero. So the weight of the host is also altered.
Otherwise the second rebalancing occured due to altered host weight after destroying osd. Setting osd out on it´s own does not alter the host weight (iirc).

That makes an odd sort of sense, and sounds like a bug, or something that should be warned about in the UI somewhere. Anyway, I know what to do next time.