keyring: (2) No such file or directory

[freakits_jino]

Hello, So We are having a 3 node cluster up and running in productions. One of our node went down and when it got back up everything was back to normal. But the only thing that was an issue is one OSD on this node was showing down/out under ceph > OSD. The HDD in reference to the osd is working fine as there are no hardware errors.

We tried starting the osd from GUI, the task runs but the osd did not come up. On checking the ceph logs we found the below error
/var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory.

When we navigated to the dir: /var/lib/ceph/osd/ceph-15 the directory was empty with no files.
While the other osds dirs have these files: block ceph_fsid fsid keyring ready require_osd_release type whoami

I and attaching the error logs for reference.

Kindly suggest how we can fix this issue. Thankyou.

 

[franciscopaniskaseker]

i am having same error after a reboot. did you find any solution?

 

[freakits_jino]

i am having same error after a reboot. did you find any solution?

Happened when we tired to hot swap a faulty HDD and an incorrect one was pulled out. the drive letter was changed and hence OSD was unable to detect. I simply deleted the OSD which went down and created it again with the below steps.

Set OSD flags for the pool does not start rebalancing .


on GUI:
locate the OSD and mark it as down and out, and then destroy it(destroy may throw an error).

If unable to perform the same from GUI then ssh to one of the cluster node and:

#ceph osd out {osd-num}
#sudo systemctl stop ceph-osd@{osd-num}
#ceph osd crush remove {name}
#ceph auth del osd.{osd-num}
#ceph osd rm {osd-num}

Lookup.Figureout the VG name of that particular OSD
#ceph-volume lvm list

Remove volume group
#vgremove -y {VG-NAME}

#ceph-volume lvm zap --destroy {DEVICE-NAME}

After above steps goto GUI and simply create/add new OSD and you are good to go.

NOTE:
1. make sure you set your ceph under maintenance by setting required flags.
2. Unset the flags post restoring the OSD
3. The ceph pool might rebalance, so keep the same under consideration before you perform the activity.