ceph osd down/out

[RobFantini]

Hello,
I have 2 osd's listed as down/out in pve > ceph > osd

for one when i press 'start' i get this,

 === osd.5 === create-or-move updated item name 'osd.5' weight 1.82 at location {host=ceph1-ib,root=default} to crush map
Starting Ceph osd.5 on ceph1-ib...
starting osd.5 at :/0 osd_data /var/lib/ceph/osd/ceph-5 /var/lib/ceph/osd/ceph-5/journal
TASK OK

However the osd does not start and is still listed as down/out


for the other osd i get this:

[COLOR=#000000][FONT=tahoma]/etc/init.d/ceph: osd.6 not found (/etc/pve/ceph.conf defines osd.7 osd.5 osd.4 mon.0 mon.1 mon.2 mon.3, /var/lib/ceph defines osd.7 osd.5 osd.4)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK ERROR: command 'setsid service ceph -c /etc/pve/ceph.conf start osd.6' failed: exit code 1[/FONT][/COLOR]

any suggestions on how to fix?

 

[udo]

Hello,
I have 2 osd's listed as down/out in pve > ceph > osd

for one when i press 'start' i get this,

 === osd.5 === create-or-move updated item name 'osd.5' weight 1.82 at location {host=ceph1-ib,root=default} to crush map
Starting Ceph osd.5 on ceph1-ib...
starting osd.5 at :/0 osd_data /var/lib/ceph/osd/ceph-5 /var/lib/ceph/osd/ceph-5/journal
TASK OK

However the osd does not start and is still listed as down/out


for the other osd i get this:

[COLOR=#000000][FONT=tahoma]/etc/init.d/ceph: osd.6 not found (/etc/pve/ceph.conf defines osd.7 osd.5 osd.4 mon.0 mon.1 mon.2 mon.3, /var/lib/ceph defines osd.7 osd.5 osd.4)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK ERROR: command 'setsid service ceph -c /etc/pve/ceph.conf start osd.6' failed: exit code 1[/FONT][/COLOR]

any suggestions on how to fix?

Hi,
the second message says that in your ceph.conf isn't an entry for osd.6?
Are all osds are described right in the ceph.conf?
Any futher infos if you start osd.5 again (also in the log-files):

initctl start ceph-osd id=5

Udo

 

[mo_]

/var/log/ceph/ceph-osd.5.log should tell you why its not starting

as for osd-6... it seems like you removed the /var/lib/ceph/osd/ceph-6 directory?

PS: OSDs typically arent defined via ceph.conf anymore, thats deprecated. They are now started via udev rules, to allow you to move a disk to another system and run the OSD there (amongst other things)

 

[RobFantini]

OK I'm trying to 'out' my last OSD [ I'm removing all and starting over ] .

after pressing out this displays:

Parameter verification failed. (400)
dev: property is missing and it is not optional

and no log entries get to /var/log/ceph/ceph-osd.0.log

Any suggestions to get that out and removed?

Or another way to clear all ceph config and start over?

 

[RobFantini]

also tried from cli...

tree:

ceph osd tree
# id    weight  type name       up/down reweight
-1      1.82    root default
-2      1.82            host ceph4-ib
0       1.82                    osd.0   down    1
-3      0               host ceph3-ib
-4      0               host ceph2-ib
-5      0               host ceph1-ib
-6      0               host dell2-ib
12      0       osd.12  down    0

out

 ceph osd out osd.0
marked out osd.0. 
ceph4-ib  /etc/pve # ceph osd tree
# id    weight  type name       up/down reweight
-1      1.82    root default
-2      1.82            host ceph4-ib
0       1.82                    osd.0   down    0
-3      0               host ceph3-ib
-4      0               host ceph2-ib
-5      0               host ceph1-ib
-6      0               host dell2-ib
12      0       osd.12  down    0

del

ceph auth del osd.0
updated

yet tree show osd.0 still there:

# id    weight  type name       up/down reweight
-1      1.82    root default
-2      1.82            host ceph4-ib
0       1.82                    osd.0   down    0
-3      0               host ceph3-ib
-4      0               host ceph2-ib
-5      0               host ceph1-ib
-6      0               host dell2-ib
12      0       osd.12  down    0

 

[RobFantini]

next tried rm, but it is still there with a diff reweight. maybe because this is the last osd?

ceph osd rm osd.0
removed osd.0

# id    weight  type name       up/down reweight
-1      1.82    root default
-2      1.82            host ceph4-ib
0       1.82                    osd.0   DNE
-3      0               host ceph3-ib
-4      0               host ceph2-ib
-5      0               host ceph1-ib
-6      0               host dell2-ib