Before restarting a CEPH OSD, should you mark it as "Out"?
Marking an OSD as out, tells Ceph that this OSD should not be part of the cluster anymore. This in turn will cause a rebalancing, as Ceph will recreate the data located on that OSD on other OSDs in the cluster.
This is only something that you want to do if you need to replace the OSD or want to destroy and recreate it.
Normally you don't need to do anything if you just restart a service (OSD, MON, MGR, MDS) as OSDs have redundancy, you might see a warning for a short time until the OSD is back up, but as long as there are copies on 2 other OSDs, restarting one OSD at a time is not a problem. The same goes for monitors as you should always have enough to form a majority (min 3. and usually 3 are enough). MGR and MDS work with fallbacks on the remaining nodes. So if you stop those, another node will take over and become active.
If you plan a longer maintenance on a node where the node might be down for some time, and you want to avoid the rebalancing of data, you can set the "noout", "norecover", "norebalance" global OSD flags for that time. The "noout" should be enough, but the others are just to be on the safe side.
The default timeout is 10 minutes. If an OSD is not back up within that time, Ceph would automatically set it to out.
Once you are done, don't forget to disable these OSD flags again to give back the self-healing possibilities that Ceph has.
