[SOLVED] Ceph Client Crash once every 10 mins, every node - error calling conf_read_file

[scyto]

i am seeing about 40 lines like this once every 10 mins or so in the system log, i see no errors in the ceph log.

Sep 25 12:48:18 pve2 ceph-crash[818]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-09-24T17:24:31.398873Z_7612796b-f316-4c05-8b76-5a56aee92dea as client.crash.pve2 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 25 12:48:18 pve2 ceph-crash[818]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-09-24T17:24:31.398873Z_7612796b-f316-4c05-8b76-5a56aee92dea as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

but everything looks ok

root@pve2:~# ceph -s
  cluster:
    id:     5e55fd50-d135-413d-bffe-9d0fae0ef5fa
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum pve3-IPv6,pve2-IPv6,pve1-IPv6 (age 3d)
    mgr: pve3(active, since 26h), standbys: pve1, pve2
    mds: 1/1 daemons up, 5 standby
    osd: 6 osds: 6 up (since 3d), 6 in (since 3d)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 113 pgs
    objects: 36.13k objects, 134 GiB
    usage:   381 GiB used, 5.1 TiB / 5.5 TiB avail
    pgs:     113 active+clean
 
  io:
    client:   0 B/s rd, 1.4 MiB/s wr, 0 op/s rd, 34 op/s wr

do i need to fix anything?

 

[scyto]

the ceph.conf file is where it should be in both in /etc/ceph and /etc/pve

i saw in a search that i can be an issue with the client keyring

I see in the conf file the path looks like this

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

I am unclear what the name of the keyring should be

it implied there should be one with the name of the cluster in it, but all i see is this:

root@pve2:/etc/pve/priv# ls -la *.keyring
-rw------- 1 root www-data 151 Aug 20 07:59 ceph.client.admin.keyring
-rw------- 1 root www-data 228 Aug 20 08:01 ceph.mon.keyring

Do i have all the keyrings i should have?
Are the attributes and guid/uid right?

 

[scyto]

easy fix, delete the crash files with rm-r /var/lib/ceph/crash/2023*

I assume it is this https://bugzilla.proxmox.com/show_bug.cgi?id=4759 preventing the ceph-crash process from collecting everything it needs when 'posting' (i assume it sends it somewhere?)