[SOLVED] 2 stuck OSD's in ceph database

[lifeboy]

I tried to remove all OSD's from a cluster and recreate them, but 2 of them are still stuck in the ceph configuration database.

I have done all the standard commands to remove them, but the reference stays.

# ceph osd crush remove osd.1
removed item id 1 name 'osd.1' from crush map
# ceph osd crush remove osd.0
removed item id 0 name 'osd.0' from crush map
# ceph auth del osd.1
# ceph auth del osd.0
# ceph osd rm 1
removed osd.1
# ceph osd rm 0
removed osd.0

The reference to these does doesn't exist in cepf.conf anymore either and neither are there directories in which they exists.

I see there are commands to dump the database:

# ceph config dump
WHO     MASK  LEVEL     OPTION                                 VALUE       RO
global        advanced  mon_allow_pool_size_one                true        
mon           advanced  auth_allow_insecure_global_id_reclaim  false        
osd.0         basic     osd_mclock_max_capacity_iops_hdd       209.684252  
osd.1         basic     osd_mclock_max_capacity_iops_hdd       212.458536

But none that will delete entries from the database?

How do I remove these two osd's?

 

[Hannes Laimer]

Hey,

did you do pveceph osd destroy <OSD>? Are the services still running systemctl status ceph-osd@0? Usually you'd do something like

ceph osd out <ODS>
systemctl stop ceph-osd@<OSD>
pveceph osd destroy <OSD>

# clear partition table
ceph-volume lvm zap /dev/sdX --destroy

then you can recreate the OSD on the previously used disk.

 

[lifeboy]

did you do pveceph osd destroy <OSD>?

That just hangs, since the osd's were on a node that doesn't exist anymore.

Are the services still running systemctl status ceph-osd@0? Usually you'd do something like

ceph osd out <ODS>
systemctl stop ceph-osd@<OSD>
pveceph osd destroy <OSD>
[/QUOTE]

Here is also says
:~# pveceph osd destroy 1
OSD osd.1 does not belong to node pmx2! at /usr/share/perl5/PVE/API2/Ceph/OSD.pm line 952, <DATA> line 960.


[QUOTE="Hannes Laimer, post: 671775, member: 97843"]


# clear partition table
ceph-volume lvm zap /dev/sdX --destroy

then you can recreate the OSD on the previously used disk.

This zapped the osd's, but they are still shown in the ceph configuration database.

 

[lifeboy]

I recreated the manager on a node (after deleting all the managers) and that resolved the issue, so I can now add the OSD's again.