`/var/lib/ceph/osd/ceph-<ID>/keyring` is gone.

[jic5760]

ceph osd is dead and there is no keyring. Why? The keyring for a few osd on a few nodes is gone.

/var/lib/ceph/osd# find 
.
./ceph-3
./ceph-3/lockbox.keyring
./ceph-2
./ceph-2/lockbox.keyring

LOG :
Jan 21 22:53:30 pve-node-stor-08101 systemd[1]: Starting Ceph object storage daemon osd.2...
Jan 21 22:53:30 pve-node-stor-08101 systemd[1]: Started Ceph object storage daemon osd.2.
Jan 21 22:53:30 pve-node-stor-08101 ceph-osd[1463084]: 2022-01-21T22:53:30.250+0900 7f87c7430f00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory
Jan 21 22:53:30 pve-node-stor-08101 ceph-osd[1463084]: 2022-01-21T22:53:30.250+0900 7f87c7430f00 -1 AuthRegistry(0x5621eb1b8a40) no keyring found at /var/lib/ceph/osd/ceph-2/keyring, disabling cephx
Jan 21 22:53:30 pve-node-stor-08101 ceph-osd[1463084]: 2022-01-21T22:53:30.250+0900 7f87c7430f00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory
Jan 21 22:53:30 pve-node-stor-08101 ceph-osd[1463084]: 2022-01-21T22:53:30.250+0900 7f87c7430f00 -1 AuthRegistry(0x7ffe5ec8e6c0) no keyring found at /var/lib/ceph/osd/ceph-2/keyring, disabling cephx
Jan 21 22:53:30 pve-node-stor-08101 ceph-osd[1463084]: failed to fetch mon config (--no-mon-config to skip)
Jan 21 22:53:30 pve-node-stor-08101 systemd[1]: ceph-osd@2.service: Main process exited, code=exited, status=1/FAILURE

 

[jic5760]

Solved

# /usr/sbin/ceph-volume-systemd lvm-{osd_id}-{lvm_name(?)}
# systemctl start ceph-osd@{osd_id}

 

[franciscopaniskaseker]

Happened with me today. I did not find the ceph-volume-systemd command. I have exacty same errors after reboot.

 

[fcarucci]

I'm having the same issue, can someone please detail the solution? Thanks!

 

[aaron]

That directory is located inside the LVM of the OSD and is mounted as a tmpfile. It first needs to be made available. If that failed for some reason, you could run ceph-volume lvm activate --all
It will scan for existing OSDs and will prepare and start them.