[SOLVED] Ceph Client Crash once every 10 mins, every node - error calling conf_read_file

scyto

Active Member
Aug 8, 2023
310
55
28
i am seeing about 40 lines like this once every 10 mins or so in the system log, i see no errors in the ceph log.

Code:
Sep 25 12:48:18 pve2 ceph-crash[818]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-09-24T17:24:31.398873Z_7612796b-f316-4c05-8b76-5a56aee92dea as client.crash.pve2 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 25 12:48:18 pve2 ceph-crash[818]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-09-24T17:24:31.398873Z_7612796b-f316-4c05-8b76-5a56aee92dea as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

but everything looks ok


Code:
root@pve2:~# ceph -s
  cluster:
    id:     5e55fd50-d135-413d-bffe-9d0fae0ef5fa
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum pve3-IPv6,pve2-IPv6,pve1-IPv6 (age 3d)
    mgr: pve3(active, since 26h), standbys: pve1, pve2
    mds: 1/1 daemons up, 5 standby
    osd: 6 osds: 6 up (since 3d), 6 in (since 3d)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 113 pgs
    objects: 36.13k objects, 134 GiB
    usage:   381 GiB used, 5.1 TiB / 5.5 TiB avail
    pgs:     113 active+clean
 
  io:
    client:   0 B/s rd, 1.4 MiB/s wr, 0 op/s rd, 34 op/s wr

do i need to fix anything?
 
Last edited:
the ceph.conf file is where it should be in both in /etc/ceph and /etc/pve

i saw in a search that i can be an issue with the client keyring

I see in the conf file the path looks like this


Code:
[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

I am unclear what the name of the keyring should be

it implied there should be one with the name of the cluster in it, but all i see is this:

Code:
root@pve2:/etc/pve/priv# ls -la *.keyring
-rw------- 1 root www-data 151 Aug 20 07:59 ceph.client.admin.keyring
-rw------- 1 root www-data 228 Aug 20 08:01 ceph.mon.keyring

Do i have all the keyrings i should have?
Are the attributes and guid/uid right?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!