failed to load /var/lib/ceph/mds/ceph-admin/keyring:

[maxellju]

Hello community ,

I had set up a Ceph cluster with three Nodes. The MDS daemon is not comming up



What I had set up


ceph auth get-or-create mds.${3} mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-${3}/keyring


# create filesystem pools
ceph osd pool create cephfs0_data
ceph osd pool create cephfs0_metadata

# create filesystem with explicit pools
ceph fs new cephfs0 cephfs0_metadata cephfs0_da




My environment

[root@node3 mnt]# ceph -s
cluster:
id: 1a15bbb4-b62c-441b-889b-4e5a230911d1
health: HEALTH_ERR
1 pools have many more objects per pg than average
2 filesystems are offline
2 filesystems are online with fewer MDS than max_mds
2 daemons have recently crashed

services:
mon: 3 daemons, quorum node1,node2,node3 (age 2h)
mgr: node3(active, since 2h), standbys: node2, node1
mds: cephfs:1 cephfs-3:0
osd: 108 osds: 108 up (since 2h), 108 in (since 8M)
rgw: 1 daemon active (node1.rgw0)

task status:

data:
pools: 28 pools, 817 pgs
objects: 102.46k objects, 36 GiB
usage: 237 GiB used, 982 TiB / 982 TiB avail
pgs: 817 active+clean



I am getting theses erros

2022-05-04T10:47:29.823-0700 7f290fd8d600 -1 auth: error reading file: /var/lib/ceph/mds/ceph-admin/keyring: bufferlist::read_file(/var/lib/ceph/mds/ceph-admin/keyring): read error21) Is a directory
2022-05-04T10:47:29.823-0700 7f290fd8d600 -1 auth: failed to load /var/lib/ceph/mds/ceph-admin/keyring: (21) Is a directory
2022-05-04T10:47:29.824-0700 7f290fd8d600 -1 auth: error reading file: /var/lib/ceph/mds/ceph-admin/keyring: bufferlist::read_file(/var/lib/ceph/mds/ceph-admin/keyring): read error21) Is a directory
2022-05-04T10:47:29.824-0700 7f290fd8d600 -1 auth: failed to load /var/lib/ceph/mds/ceph-admin/keyring: (21) Is a directory
2022-05-04T10:47:29.824-0700 7f290fd8d600 -1 auth: error reading file: /var/lib/ceph/mds/ceph-admin/keyring: bufferlist::read_file(/var/lib/ceph/mds/ceph-admin/keyring): read error21) Is a directory
2022-05-04T10:47:29.824-0700 7f290fd8d600 -1 auth: failed to load /var/lib/ceph/mds/ceph-admin/keyring: (21) Is a directory
2022-05-04T10:47:29.824-0700 7f290fd8d600 -1 monclient: keyring not found
failed to fetch mon config (--no-mon-config to skip)

[root@node3 mnt]# ceph-mds -1 auth
2022-05-04T10:49:07.396-0700 7f9d6672a600 -1 auth: error reading file: /var/lib/ceph/mds/ceph-admin/keyring: bufferlist::read_file(/var/lib/ceph/mds/ceph-admin/keyring): read error21) Is a directory
2022-05-04T10:49:07.396-0700 7f9d6672a600 -1 auth: failed to load /var/lib/ceph/mds/ceph-admin/keyring: (21) Is a directory
2022-05-04T10:49:07.397-0700 7f9d6672a600 -1 auth: error reading file: /var/lib/ceph/mds/ceph-admin/keyring: bufferlist::read_file(/var/lib/ceph/mds/ceph-admin/keyring): read error21) Is a directory
2022-05-04T10:49:07.397-0700 7f9d6672a600 -1 auth: failed to load /var/lib/ceph/mds/ceph-admin/keyring: (21) Is a directory
2022-05-04T10:49:07.397-0700 7f9d6672a600 -1 auth: error reading file: /var/lib/ceph/mds/ceph-admin/keyring: bufferlist::read_file(/var/lib/ceph/mds/ceph-admin/keyring): read error21) Is a directory
2022-05-04T10:49:07.397-0700 7f9d6672a600 -1 auth: failed to load /var/lib/ceph/mds/ceph-admin/keyring: (21) Is a directory
2022-05-04T10:49:07.397-0700 7f9d6672a600 -1 monclient: keyring not found
failed to fetch mon config (--no-mon-config to skip)

[root@node3 mnt]# ceph-mds -i auth
2022-05-04T10:49:16.889-0700 7f0107ff3600 -1 auth: unable to find a keyring on /var/lib/ceph/mds/ceph-auth/keyring: (2) No such file or directory
2022-05-04T10:49:16.889-0700 7f0107ff3600 -1 AuthRegistry(0x556d52556940) no keyring found at /var/lib/ceph/mds/ceph-auth/keyring, disabling cephx
2022-05-04T10:49:16.889-0700 7f0107ff3600 -1 auth: unable to find a keyring on /var/lib/ceph/mds/ceph-auth/keyring: (2) No such file or directory
2022-05-04T10:49:16.889-0700 7f0107ff3600 -1 AuthRegistry(0x7ffe7c2242b0) no keyring found at /var/lib/ceph/mds/ceph-auth/keyring, disabling cephx
failed to fetch mon config (--no-mon-config to skip)
[root@node3 mnt]# ^C