[SOLVED] adding back previously removed osd goes wrong

[walter.egosson]

Hi,
For some reason, we removed an osd and decided to add it back under the same id (removed osd.6 then added it back under osd.6)
We followed the ceph documentation: ceph documentation

The problem is, the osd is not working anymore when we added it back

1/ the osd is marked as down (ceph osd tree):

6 hdd 0.90919 osd.6 down 0 1.00000

2/ it shows an weird row when running ceph osd status

+----+------------+-------+-------+--------+---------+--------+---------+------------+
| id | host | used | avail | wr ops | wr data | rd ops | rd data | state |
+----+------------+-------+-------+--------+---------+--------+---------+------------+
| 0 | xxxxxxxxx | 390G | 442G | 8 | 98.4k | 1 | 90 | exists,up |
| 1 | xxxxxxxxx| 421G | 411G | 2 | 120k | 0 | 0 | exists,up |
| 2 | xxxxxxxxx| 235G | 598G | 5 | 58.4k | 0 | 0 | exists,up |
| 3 | xxxxxxxxx | 279G | 553G | 6 | 88.8k | 0 | 0 | exists,up |
| 4 | xxxxxxxxx | 248G | 585G | 2 | 20.8k | 0 | 0 | exists,up |
| 5 | xxxxxxxxx | 224G | 609G | 1 | 28.0k | 0 | 0 | exists,up |
| 6 | | 0 | 0 | 0 | 0 | 0 | 0 | exists,new |
| 7 | xxxxxxxxx| 279G | 651G | 5 | 60.9k | 0 | 0 | exists,up |
| 8 | xxxxxxxxx | 318G | 613G | 2 | 93.6k | 1 | 16 | exists,up |
+----+------------+-------+-------+--------+---------+--------+---------+------------+

3/ Neither /etc/init.d/ceph -a start osd.6 and systemctl start osd.6 return error and both show osd.6 as running

Thanks!!

 

[fiona]

Hi,
were you able to resolve the issue in the meantime?

Otherwise I'd suggest trying to remove the OSD (after doing out and stop) via the GUI and mark the Cleanup Disks checkbox. Doing it via the GUI (or PVE API if you want) there's less chance to miss something and I'm not fully sure there is no PVE-related config that might be left-over when you do the manual steps from the ceph documentation. Afterwards re-add the drive as a fresh OSD.

 

[walter.egosson]

sorry for the late response,
added back via the GUI as you said solved the issue, it looks like GUI-way beat the console this time