Ceph can't start OSD

[Red]

Hi to all,
I've an enormous problem with ceph.
This is my configuration:
[global]

auth client required = cephx
auth cluster required = cephx
auth service required = cephx c
luster network = 10.10.10.0/30
fsid = 3f50371b-9be3-4af3-8e6d-c42e06ab3d22
keyring = /etc/pve/priv/$cluster.$name.keyring
mon allow pool delete = true
osd journal size = 5120
osd pool default min size = 2
osd pool default size = 3
public network = 10.10.10.0/30​

[osd]

keyring = /var/lib/ceph/osd/ceph-$id/keyring​

[mon.zero]

host = zero mon addr = 10.10.10.1:6789​

Till about 2 or 3 hours ago, everything was going well, but somehow,
ceph crashed and an osd has failed.
I have 2 OSD, 1T each:


I don't know what the hell happend, but I this log from ceph:

root@zero:/var/log/ceph# tail -10f ceph-mgr.zero.log
2018-03-11 22:54:59.505170 7ff16fc6f700 1 mgr send_beacon active
2018-03-11 22:54:59.505423 7ff16fc6f700 0 Cannot get stat of OSD 1
2018-03-11 22:55:01.505722 7ff16fc6f700 1 mgr send_beacon active
2018-03-11 22:55:01.506026 7ff16fc6f700 0 Cannot get stat of OSD 1

root@zero:/var/log/ceph# tail -10f ceph-osd.1.log
2018-03-11 22:44:53.007227 7f24d94b8e00 1 bdev(0x555d4fd3eb40 /var/lib/ceph/osd/ceph-1/block) open path /var/lib/ceph/osd/ceph-1/block
2018-03-11 22:44:53.007378 7f24d94b8e00 1 bdev(0x555d4fd3eb40 /var/lib/ceph/osd/ceph-1/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational
2018-03-11 22:44:53.007524 7f24d94b8e00 1 bdev(0x555d4fd3eb40 /var/lib/ceph/osd/ceph-1/block) close
2018-03-11 22:44:53.317214 7f24d94b8e00 1 bluestore(/var/lib/ceph/osd/ceph-1) _mount path /var/lib/ceph/osd/ceph-1
2018-03-11 22:44:53.317411 7f24d94b8e00 1 bdev create path /var/lib/ceph/osd/ceph-1/block type kernel
2018-03-11 22:44:53.317425 7f24d94b8e00 1 bdev(0x555d4fd3ed80 /var/lib/ceph/osd/ceph-1/block) open path /var/lib/ceph/osd/ceph-1/block
2018-03-11 22:44:53.317524 7f24d94b8e00 1 bdev(0x555d4fd3ed80 /var/lib/ceph/osd/ceph-1/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational
2018-03-11 22:44:53.317643 7f24d94b8e00 1 bdev(0x555d4fd3ed80 /var/lib/ceph/osd/ceph-1/block) close
2018-03-11 22:44:53.616917 7f24d94b8e00 -1 osd.1 0 OSD:init: unable to mount object store
2018-03-11 22:44:53.616939 7f24d94b8e00 -1 ** ERROR: osd init failed: (2) No such file or directory


Can someone help?
I'm quite disperate... i've just finished a lot of work and now it all lost, because it crashed while I was doing backups....

 

[Alwin]

osd pool default min size = 2
osd pool default size = 3

You have min_size 2, this makes sure that the cluster has two copies of the object available and it stops writing to the pool when it is below the min_size.

How many monitors and OSDs do you have? And what is in your crushmap? Maybe you can simply replace the OSD (easiest case) or you need to check why the small xfs partition couldn't be mounted.

 

[Red]

Hi Alwin,
I've just one monitor, because my server it's KVM and storage (it's a test enviroment quite stupid),
How can I check why he can't mount the osd?

If I set the min size to 1 temporanely, can I try to restore my data?

Thanks
Red

Edit: I've in total 2 OSD on my server

 

[Alwin]

If I set the min size to 1 temporanely, can I try to restore my data?

If you didn't change the crush rule to distribute data on OSD level, then there is no complete set of data available. You then either need backups or try to get the xfs partition to mount. Then you can check if there are subsequent errors on the DB/WAL/data part of the OSD.

 

[Red]

Here's my crush map:
begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 1 osd.1 class hdd
device 4 osd.4 class hdd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host zero {

id -3 # do not change unnecessarily
id -2 class hdd # do not change unnecessarily
# weight 1.819
alg straw2
hash 0 # rjenkins1
item osd.1 weight 0.910
item osd.4 weight 0.910​

}
root default {

id -1 # do not change unnecessarily
id -4 class hdd # do not change unnecessarily
# weight 1.819
alg straw2
hash 0 # rjenkins1
item zero weight 1.819​

}

# rules rule
replicated_rule {

id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit​

}
# end crush map

try to get the xfs partition to mount. Then you can check if there are subsequent errors on the DB/WAL/data part of the OSD.

How can I try to mount it? What should it be it's mount point?

 

[Alwin]

# rules rule
replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit }
# end crush map

The part with "type host" tells ceph to distribute its copies on host level, so that a host can fail and not hold more then one copy of a object.

You should find clou's in the syslog/dmesg and try to mount the filesystem by hand.