CEPH OSD FLAGS
noout -- If the mon osd report timeout is exceeded and an OSD has not reported to the monitor, the OSD will get marked out. The “noout” flag tells the ceph monitors not to “out” any OSDs from the crush map and not to start recovery and re-balance activities, to maintain the replica count.
nobackfill -- If you need to take an OSD or node down temporarily, (e.g., upgrading daemons), you can set nobackfill so that Ceph will not backfill while the OSD(s) is down.
norecover -- Ceph will prevent new recovery operations. If you need to replace an OSD disk and don’t want the PGs to recover to another OSD while you are hotswapping disks, you can set norecover to prevent the other OSDs from copying a new set of PGs to other OSDs.
norebalance -- data rebalancing is suspended
nodown -- Prevent OSDs from getting marked down. Networking issues may interrupt Ceph heartbeat processes, and an OSD may be up but still get marked down. You can set nodown to prevent OSDs from getting marked down while troubleshooting the issue. If something (like network issue,etc) is causing OSDs to ‘flap’ (repeatedly getting marked down and then up again), you can force the monitors to stop the flapping by temporarily freezing their states with nodown.
pause -- Ceph will stop processing read and write operations, but will not affect OSD in, out, up or down statuses. If you need to troubleshoot a running Ceph cluster without clients reading and writing data, you can set the cluster to pause to prevent client operations.
Try setting ceph flags as per your requirements before rebooting a node in the cluster. works like a charm.
# Node maintenance
# stop and wait for scrub and deep-scrub operations
ceph osd set noscrub
ceph osd set nodeep-scrub
ceph status
# set cluster in maintenance mode with : (I had used the below to bring the entire cluster down when we were physically migarting the entire setup to a diffrent datacentre)
ceph -s (to check ceph status)
# ceph osd set noout
# ceph osd set nobackfill
# ceph osd set norecover
# ceph osd set norebalance
# ceph osd set nodown
# ceph osd set pause
UNSET FLAGS ONCE ACTIVITY IS COMPLETED.
