SSD for OSDs in my ceph cluster is reaching it's "write" life span. can I just clone it?

[nttec]

Can I clone my db SSD that I use for the OSDs in my ceph cluster? It is reaching it's "write" life span.

 

[aaron]

Never tried it, and it is definitely a "your mileage may vary" situation. Ideally, and that is the beauty of Ceph, you just recreate those OSDs with the new DB SSD. Ceph will do some rebalancing but if you have enough OSDs and Nodes in your cluster, you will never have a reduced redundancy situation.

Ideally, you first set the affected OSDs to out. Wait for Ceph to recreate the data that is on those OSDs somewhere else in the cluster. Once Ceph reports a Health_OK you can stop and destroy those OSDs.
Then recreate them with the new DB SSD and Ceph will rebalance again.

 

[nttec]

Never tried it, and it is definitely a "your mileage may vary" situation. Ideally, and that is the beauty of Ceph, you just recreate those OSDs with the new DB SSD. Ceph will do some rebalancing but if you have enough OSDs and Nodes in your cluster, you will never have a reduced redundancy situation.

Ideally, you first set the affected OSDs to out. Wait for Ceph to recreate the data that is on those OSDs somewhere else in the cluster. Once Ceph reports a Health_OK you can stop and destroy those OSDs.
Then recreate them with the new DB SSD and Ceph will rebalance again.

Thank you for your reply. Have a nice day.

 

[michael.schaefers]

if you have the physical space for adding the new SSD _before_ removing the old one, you could even add a new OSD and set the affected OSD out (not down!) and wait for ceph to rebalance. That would save you the second rebalancing while maintaining full redundancy during the operation

 

[nttec]

if you have the physical space for adding the new SSD _before_ removing the old one, you could even add a new OSD and set the affected OSD out (not down!) and wait for ceph to rebalance. That would safe you from the second rebalancing while maintaining full redundancy during the operation

will keep this in mind. thank you for your recommendation.

 

[nttec]

Never tried it, and it is definitely a "your mileage may vary" situation. Ideally, and that is the beauty of Ceph, you just recreate those OSDs with the new DB SSD. Ceph will do some rebalancing but if you have enough OSDs and Nodes in your cluster, you will never have a reduced redundancy situation.

Ideally, you first set the affected OSDs to out. Wait for Ceph to recreate the data that is on those OSDs somewhere else in the cluster. Once Ceph reports a Health_OK you can stop and destroy those OSDs.
Then recreate them with the new DB SSD and Ceph will rebalance again.

@aaron I have 15 OSD. 5 OSD on each SSD and I believe the redundancy is set as 3/2. Better to remove 1 OSD at a time or all 5 OSD that is on the SSD? To be more clear 3 ceph host, and 5 OSD on each host.

@michael.schaefers - any thoughts?

 

[nttec]

we proceed with installing additional ssd one at a time and now my additional question is:

Can we set the no rebalance and no backfill flag ,destroy the OSD and readd the OSD, unset the flags and have it rebuild.
to prevent multiple rebuilds each time.

 

[aaron]

I would also set the "norecovery" flag. Keep an eye on the cluster because if another OSD fails (let's hope not, but you never know), you are likely to have at least some PGs with only one copy left -> IO blocked for the affected pools. In that situation, you should disable the flags to let Ceph recover and hopefully get you back to 2 copies fast to have IO working again.