ceph-crash problem

radim.smehlik · Jun 5, 2023

Hello,

after upgrading Ceph packages from version 17.2.5 to 17.2.6, the syslog contains following warnings:

Jun 05 09:20:24 pve01 ceph-crash[1883176]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-06-01T19:33:42.133764Z_6d5e3451-8d05-4932-8b5a-a86dca582440 as client.crash.pve01 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jun 05 09:20:24 pve01 ceph-crash[1883176]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-06-01T19:33:42.133764Z_6d5e3451-8d05-4932-8b5a-a86dca582440 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jun 05 09:20:24 pve01 ceph-crash[1883176]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-06-01T19:33:42.133764Z_6d5e3451-8d05-4932-8b5a-a86dca582440 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

There are probably some problems with the permission settings after upgrading to the latest Ceph packages. The Proxmox Ceph GUI shows no errors, but there are some unprocessed crash report files in /var/lib/ceph/crash.

Thank you for the answer.

radim.smehlik · Jun 6, 2023

Another point to this problem. From version 17.2.6, there is a new function in /usr/bin/ceph-crash script (drop_privs()), that changes user to ceph. This user does not have permissions to read /etc/ceph/ceph.conf (linked to /etc/pve/ceph.conf 640 root:www-data) and /etc/pve/priv dir, that contains keyrings. When we temporary comment out this function, that drops root privileges, everything work as expected.

Changing ceph.conf permissions is not permitted.
root@pve01:/etc/pve# chmod 644 ceph.conf
chmod: changing permissions of 'ceph.conf': Operation not permitted

We do not have ceph-mgr-dashboard package installed as there is no reason for this. So this is probably not the same problem like in this thread.

radim.smehlik · Jun 6, 2023

There will be fix in the next Ceph release. Bug 4759.

itobin · Jul 3, 2023

we are also seeing this.

flotho · Jul 15, 2023

also impacted on latest proxmox 8.0.3

radim.smehlik · Jul 17, 2023

Unfortunately, because the problem with file permissions is more complicated, the bug was reopened.

Mario Hosse · Jul 20, 2023

We have the same problem after release upgrade to Proxmox 8.0.3 and ceph 17.2.6. Every 10 minutes in syslog on each node a message. On another cluster with Proxmox 7.4.15 and a ceph 17.2.6 there is not this problem.

ITT · Jul 20, 2023

Same here.

plnt · Jul 28, 2023

Hi,

in what time horizon is the bug fixed?

TY
Andrej

gio2022 · Jul 31, 2023

Same error.
pve-manager/8.0.3/bbf3993334bfa916 (running kernel: 6.2.16-5-pve)

flotho · Jul 31, 2023

plnt said:
Hi,

in what time horizon is the bug fixed?

TY
Andrej

Hi,
It depends on the ceph team, not the proxmox one

radim.smehlik · Aug 7, 2023

flotho said:
Hi,
It depends on the ceph team, not the proxmox one

HI, I don't think so. The Proxmox team have to solve it, because they changed ceph's configuration file to a symbolic link (in the past), which points to a /etc/pve/ceph.conf and that is not readable by the Ceph user.

You can read the comments in the Bugzilla.

flotho · Aug 9, 2023

Hi @radim.smehlik ,
I understand your point of view. Yet, AFAICS, the ceph bug talk about the same issue, without proxmox.... So IMHO, ceph should first solve the issue or maybe Proxmox team could propose a solution to the ceph team. It looks much mainstream to me this way.
But yeah.... we all expecting this to be solved ASAP ;-)

mattkenn4545 · Sep 6, 2023

I'm just running into this as well. Doesn't look like there has been any movement on the bug or this thread. Is there an ETA for resolution?

cne-solutions · Sep 6, 2023

We are facing the same behavior. I disabled the ceph-dashboard, but it won`t fix the issue.

godsavethequ33n · Sep 6, 2023

I am noticing this as well:

Code:

Sep 06 16:43:10 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.crash.pm failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 06 16:43:10 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 06 16:43:10 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 06 16:53:11 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.crash.pm failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 06 16:53:11 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 06 16:53:11 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 06 17:03:11 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.crash.pm failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 06 17:03:11 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Sep 06 17:03:11 pm ceph-crash[7542]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

Code:

 pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.111-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-5
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.111-1-pve: 5.15.111-1
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.13.19-6-pve: 5.13.19-15
ceph: 17.2.6-pve1
ceph-fuse: 17.2.6-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.3-1
proxmox-backup-file-restore: 2.4.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-5
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

high_performer · Sep 13, 2023

Are these "crash-files" of any importance? They are somewhat old. Yes - there seemed to be some problems. But the number of files does not increase.
Is it risky to simply delete them?

radim.smehlik · Sep 13, 2023

Hi. You can get more information in this thread or in the official ceph documentation.

ITT · Sep 13, 2023

high_performer said:
Are these "crash-files" of any importance? They are somewhat old. Yes - there seemed to be some problems. But the number of files does not increase.
Is it risky to simply delete them?

So i deleted these annoying things and it´s silent now.

godsavethequ33n · Sep 14, 2023

ITT said:
So i deleted these annoying things and it´s silent now.

So in my example

Code:

/var/lib/ceph/crash/2023-08-23T04:12:26.150675Z_44d4035b-5ea9-4452-93c3-22a06b877464

is a directory.. did you just remove the whole directory?

ceph-crash problem

Member

Member

Member

Well-Known Member

Renowned Member

Member

Well-Known Member

Well-Known Member

Member

Member

Renowned Member

Member

Renowned Member

Renowned Member

Member

Member

Active Member

Member

Well-Known Member

Member