[CEPH] auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied

[fcarucci]

Hello, I'm pretty new to Ceph, I have a 3-node cluster.
On one of the nodes I keep getting these warnings in the log:

Mar 07 14:59:02 pve-ceph1 ceph-crash[974]: WARNING:ceph-crash:post /var/lib/ceph/crash/2024-03-06T03:47:35.113588Z_ae2458c5-5cb9-4b85-9afe-1b2efe7f8600 as client.admin failed: 2024-03-07T14:59:02.088-0800 7639492346c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Mar 07 14:59:02 pve-ceph1 ceph-crash[974]: 2024-03-07T14:59:02.096-0800 7639492346c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Mar 07 14:59:02 pve-ceph1 ceph-crash[974]: 2024-03-07T14:59:02.096-0800 7639492346c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Mar 07 14:59:02 pve-ceph1 ceph-crash[974]: 2024-03-07T14:59:02.096-0800 7639492346c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Mar 07 14:59:02 pve-ceph1 ceph-crash[974]: 2024-03-07T14:59:02.096-0800 7639492346c0 -1 monclient: keyring not found

The keyring file is there:

root@pve-ceph1:~# ls -la /etc/pve/priv/ceph.client.admin.keyring
-rw------- 1 root www-data 151 Feb 29 14:22 /etc/pve/priv/ceph.client.admin.keyring

The cluster seems otherwise fine and healthy.

How do I clear this warnings? What kind of information do you I need to diagnose the problem?

Thanks!

 

[pille99]

show your ceph.conf plz

 

[fcarucci]

Thanks for your help!

[global]
         auth_client_required = cephx
         auth_cluster_required = cephx
         auth_service_required = cephx
         cluster_network = 10.0.20.4/24
         err_to_syslog = true
         fsid = 5ce42d57-4371-475a-94fb-eac8acefe72e
         mon_allow_pool_delete = true
         mon_allow_pool_size_one = false
         mon_cluster_log_file_level = info
         mon_cluster_log_to_file = false
         mon_host = 10.0.20.3 10.0.20.1 10.0.20.4
         ms_bind_ipv4 = true
         ms_bind_ipv6 = false
         osd_deep_scrub_interval = 1209600
         osd_pool_default_min_size = 2
         osd_pool_default_size = 4
         osd_scrub_begin_hour = 23
         osd_scrub_end_hour = 7
         osd_scrub_sleep = 0.1
         public_network = 10.0.20.4/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
         keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.pve-1]
         host = pve
         mds_standby_for_name = pve

[mds.pve-2]
         host = pve
         mds_standby_for_name = pve

[mds.pve-3]
         host = pve
         mds_standby_for_name = pve

[mds.pve-ceph1-1]
         host = pve-ceph1
         mds_standby_for_name = pve

[mds.pve-ceph1-2]
         host = pve-ceph1
         mds_standby_for_name = pve

[mds.pve-ceph1-3]
         host = pve-ceph1
         mds_standby_for_name = pve

[mds.pve2-1]
         host = pve2
         mds_standby_for_name = pve

[mds.pve3-1]
         host = pve3
         mds_standby_for_name = pve

[mon.pve]
         debug_mon = 0/5
         public_addr = 10.0.20.1

[mon.pve-ceph1]
         public_addr = 10.0.20.4

[mon.pve3]
         debug_mon = 0/5
         public_addr = 10.0.20.3

Feel free to point out anything wrong in this .conf.

 

[fcarucci]

If it helps, this spam happens exactly every 10 minutes.

 

[fabian]

this is a known issue that will be fixed with one of the next versions, it's spammy but all it does is prevent crash information from being submitted upstream.

 

[fcarucci]

Thanks so much! It's benign then.

 

[flotho]

this is a known issue that will be fixed with one of the next versions, it's spammy but all it does is prevent crash information from being submitted upstream.

Hi @fabian , than ks for the answer, any chance to point the issue so that we could follow up the resolution?
Regards

 

[fabian]

https://git.proxmox.com/?p=ceph.git;a=commit;h=01e4a3cd84ed804450aa3c89e953d925e04d8278
https://git.proxmox.com/?p=ceph.git;a=commit;h=6fb7442deb71ab20e2644fad10ea8b08fc0f6de7

the pve-manager part is still in development:
https://lists.proxmox.com/pipermail/pve-devel/2024-April/062489.html

https://bugzilla.proxmox.com/show_bug.cgi?id=4759