RADOS permission denied (error connecting to the cluster)

[pille99]

what does it mean ?
i just updated from 7 to pve8
these error i can see on node 1, 3 and 4. node 2 doesnt show that messages


Feb 19 21:50:20 hvirt01 ceph-crash[971]: WARNING:ceph-crashost /var/lib/ceph/crash/2024-02-19T17:08:19.041904Z_847b484e-f39e-4843-b8d8-dd6e62d7ac8c as client.crash.hvirt01 failed: 2024-02-19T21:50:20.271+0100 7f55898ff6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.crash.hvirt01.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.271+0100 7f55898ff6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.crash.hvirt01.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.275+0100 7f55898ff6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.crash.hvirt01.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.275+0100 7f55898ff6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.crash.hvirt01.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.275+0100 7f55898ff6c0 -1 monclient: keyring not found
Feb 19 21:50:20 hvirt01 ceph-crash[971]: [errno 13] RADOS permission denied (error connecting to the cluster)
Feb 19 21:50:20 hvirt01 ceph-crash[971]: WARNING:ceph-crashost /var/lib/ceph/crash/2024-02-19T17:08:19.041904Z_847b484e-f39e-4843-b8d8-dd6e62d7ac8c as client.crash failed: 2024-02-19T21:50:20.335+0100 7f64380ff6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.crash.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.339+0100 7f64380ff6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.crash.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.339+0100 7f64380ff6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.crash.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.339+0100 7f64380ff6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.crash.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.339+0100 7f64380ff6c0 -1 monclient: keyring not found
Feb 19 21:50:20 hvirt01 ceph-crash[971]: [errno 13] RADOS permission denied (error connecting to the cluster)
Feb 19 21:50:20 hvirt01 ceph-crash[971]: WARNING:ceph-crashost /var/lib/ceph/crash/2024-02-19T17:08:19.041904Z_847b484e-f39e-4843-b8d8-dd6e62d7ac8c as client.admin failed: 2024-02-19T21:50:20.399+0100 7f94d6fae6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.403+0100 7f94d6fae6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.403+0100 7f94d6fae6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.403+0100 7f94d6fae6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Feb 19 21:50:20 hvirt01 ceph-crash[971]: 2024-02-19T21:50:20.403+0100 7f94d6fae6c0 -1 monclient: keyring not found
Feb 19 21:50:20 hvirt01 ceph-crash[971]: [errno 13] RADOS permission denied (error connecting to the cluster)

anything to do ?

 

[ness1602]

You don't have monitor on these nodes?

 

[pille99]

i have, on all 4 and they are running

here are the ceph.conf

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.10.11.10/24
     fsid = a8939e3c-7fee-484c-826f-29875927cf43
     mon_allow_pool_delete = true
     mon_host = 10.10.10.10 10.10.10.12 10.10.10.13 10.10.10.11
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 10.10.10.10/24
         debug_lockdep = 0/0
         debug_context = 0/0
         debug_crush = 0/0
         debug_buffer = 0/0
         debug_timer = 0/0
         debug_filer = 0/0
         debug_objecter = 0/0
         debug_rados = 0/0
         debug_rbd = 0/0
         debug_journaler = 0/0
         debug_objectcatcher = 0/0
         debug_client = 0/0
         debug_osd = 0/0
         debug_optracker = 0/0
         debug_objclass = 0/0
         debug_filestore = 0/0
         debug_journal = 0/0
         debug_ms = 0/0
         debug_monc = 0/0
         debug_tp = 0/0
         debug_auth = 0/0
         debug_finisher = 0/0
         debug_heartbeatmap = 0/0
         debug_perfcounter = 0/0
         debug_asok = 0/0
         debug_throttle = 0/0
         debug_mon = 0/0
         debug_paxos = 0/0
         debug_rgw = 0/0

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.hvirt01]
     public_addr = 10.10.10.10

[mon.hvirt02]
     public_addr = 10.10.10.11

[mon.hvirt03]
     public_addr = 10.10.10.12

[mon.hvirt04]
     public_addr = 10.10.10.13

 

[sb-jw]

Search the forum here, the error already existed and there was also a solution. The CEPH Crash Service does not have any permissions, but it wants them.

Otherwise you should delete one of your mons. The number of mons must be odd. You need at least 3 Mons and these are enough for several hundred OSDs, then you can go to 5 Mons at some point and you only have to go to 7 MONs in absolutely exceptional cases if you are running a very complex construct.