[SOLVED] Ceph Pacific Issue

[clumbo]

Hi,

I am delving into the world of ceph for the first time and seem to have a problem, running pacific with a 3 node cluster of nucs

May 25 19:38:50 nuc0 ceph-crash[741]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-23T22:17:58.511383Z_168f48ae-991d-46f3-9836-7103411602f6 as client.crash.nuc0 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
May 25 19:38:50 nuc0 ceph-crash[741]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-23T22:17:58.511383Z_168f48ae-991d-46f3-9836-7103411602f6 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
May 25 19:38:50 nuc0 ceph-crash[741]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-23T22:17:58.511383Z_168f48ae-991d-46f3-9836-7103411602f6 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

Any help with this problem or how to figure out what the issue is would be greatly appreciated.

Thanks

 

[gurubert]

Usually there is a symlink from /etc/pve/ceph.conf to /etc/ceph. Does this exist?

 

[clumbo]

Thanks for the reply

Yes it exists

lrwxrwxrwx 1 root root 18 May 23 22:22 /etc/ceph/ceph.conf -> /etc/pve/ceph.conf

 

[clumbo]

Ok I see whats going on.

Something is trying to use the ceph orch command.

    "backtrace": [
        "  File \"/usr/share/ceph/mgr/nfs/module.py\", line 169, in cluster_ls\n    return available_clusters(self)",
        "  File \"/usr/share/ceph/mgr/nfs/utils.py\", line 38, in available_clusters\n    completion = mgr.describe_service(service_type='nfs')",
        "  File \"/usr/share/ceph/mgr/orchestrator/_interface.py\", line 1488, in inner\n    completion = self._oremote(method_name, args, kwargs)",
        "  File \"/usr/share/ceph/mgr/orchestrator/_interface.py\", line 1555, in _oremote\n    raise NoOrchestrator()",
        "orchestrator._interface.NoOrchestrator: No orchestrator configured (try `ceph orch set backend`)"
    ],

How can I stop proxmox doing this, my understanding is proxmox should not have this enabled but something is trying use it?

 

[jsterr]

Ok I see whats going on.

Something is trying to use the ceph orch command.

    "backtrace": [
        "  File \"/usr/share/ceph/mgr/nfs/module.py\", line 169, in cluster_ls\n    return available_clusters(self)",
        "  File \"/usr/share/ceph/mgr/nfs/utils.py\", line 38, in available_clusters\n    completion = mgr.describe_service(service_type='nfs')",
        "  File \"/usr/share/ceph/mgr/orchestrator/_interface.py\", line 1488, in inner\n    completion = self._oremote(method_name, args, kwargs)",
        "  File \"/usr/share/ceph/mgr/orchestrator/_interface.py\", line 1555, in _oremote\n    raise NoOrchestrator()",
        "orchestrator._interface.NoOrchestrator: No orchestrator configured (try `ceph orch set backend`)"
    ],

How can I stop proxmox doing this, my understanding is proxmox should not have this enabled but something is trying use it?

Did you install ceph dashboard ? (its not default) might come from that?

 

[clumbo]

Did you install ceph dashboard ? (its not default) might come from that?

You are correct I have figured it out.

The dashboard was enabled on 1 of the nodes but not the mgr standby node enabling the dashboard on the standby node has fixed the issue

Thanks

 

[p.jahn]

It seems that I have the same issue since I updated to the latest PVE 7.4-3 last weekend.

Spoiler: pveversion -v

proxmox-ve: 7.4-1 (running kernel: 5.15.107-2-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-3
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.60-2-pve: 5.15.60-2
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.15.30-1-pve: 5.15.30-1
ceph: 17.2.6-pve1
ceph-fuse: 17.2.6-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-2
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

Jun 06 08:09:51 pve01 ceph-crash[3725]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-06-03T14:02:40.223217Z_2a44a29b-9280-4f6a-9275-d65f628c1699 as client.crash.pve01 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jun 06 08:09:51 pve01 ceph-crash[3725]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-06-03T14:02:40.223217Z_2a44a29b-9280-4f6a-9275-d65f628c1699 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jun 06 08:09:51 pve01 ceph-crash[3725]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-06-03T14:02:40.223217Z_2a44a29b-9280-4f6a-9275-d65f628c1699 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

The symlink exists and enabling the CEPH dashboard on all mgr nodes did not solve it unfortunately.

root@pve01:~# ls -al /etc/ceph/ceph.conf
lrwxrwxrwx 1 root root 18 Mar  4  2019 /etc/ceph/ceph.conf -> /etc/pve/ceph.conf

Could someone please advise how to fix this. Many Thanks.

 

[radim.smehlik]

Could someone please advise how to fix this. Many Thanks.

Look to this thread. Bug reported. Fix will be in the next Ceph release.