Replacing disk on Ceph

[Timothy1056]

Good day,

I am currently replacing disk in my ceph cluster and I have notice that when I out the OSD Ceph will start rebalancing/recovering. I Understand this part, but when the placement groups are moved off and the OSD is now safe to be destroyed and I destroy them the system starts rebalancing/recovering again see bellow procedure.

ceph osd out osd.<id>
ceph osd safe-to-destroy osd.<id>
systemctl stop ceph-osd@<id>.service
pveceph osd destroy <id>

 

[mira]

Yes, the layout changes and as a result the placement of the PGs and objects does as well.
If you don't want to have it change between destroying and adding a new OSD, you can set some global flags (GUI -> Node -> Ceph -> OSDs -> Manage Global Flags): noout, norecover, nobackfill
Remove those after you've finished adding the OSD again.

Always make sure the cluster is healthy before destroying an OSD!
Always make sure to only destroy a single OSD before letting it recover. Otherwise you may experience data loss.

 

[Timothy1056]

Hi Mira,

Thank you for the Info.