Adding new Member / ceph.conf not existing

[Opnek]

Hi,

we are currently running a Proxmox-cluster with 5 nodes.

As a storage we use Ceph.

Becuase of some capacity trouble, we planed to add a new node to the cluster.

So i installed Proxmox on the new server, configured everything and joined it to the existing Proxmox-cluster.

Everything is working as expected so far.

Before creating the OSD's on the new host, i ran the command "ceph status" on the host, which ended up with an error.

"Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)"

I switched to /etc/ceph" and the "ceph.conf" file doesnt exist, but via the Proxmox webgui ceph seems to work.

Can anybody please tell me, if it is save to just copy the "ceph.conf" from another node?

When does the host normally creates the "ceph.conf file" for the first time? Is it possible that i have forgotten an important step?

I have some concerns to completly screw up my ceph-cluster.

Any help would be appreciated.

Thanks!

 

[tim]

What steps did you take to install Ceph on the newly added node?

 

[Opnek]

I ran "pveceph install" before joining the existing cluster.

The following packages were installed on the new host:

dpkg -l | grep ceph
ii  ceph                                 14.2.9-pve1                 amd64        distributed storage and file system
ii  ceph-base                            14.2.9-pve1                 amd64        common ceph daemon libraries and management tools
ii  ceph-common                          14.2.9-pve1                 amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            14.2.9-pve1                 amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             14.2.9-pve1                 amd64        metadata server for the ceph distributed file system
ii  ceph-mgr                             14.2.9-pve1                 amd64        manager for the ceph distributed storage system
ii  ceph-mon                             14.2.9-pve1                 amd64        monitor server for the ceph storage system
ii  ceph-osd                             14.2.9-pve1                 amd64        OSD server for the ceph storage system
ii  libcephfs2                           14.2.9-pve1                 amd64        Ceph distributed file system client library
ii  python-ceph-argparse                 14.2.9-pve1                 all          Python 2 utility libraries for Ceph CLI
ii  python-cephfs                        14.2.9-pve1                 amd64        Python 2 libraries for the Ceph libcephfs library

 

[tim]

Ok next time do it after the cluster join, I guess the join did override the /etc/pve/ceph.conf file and therefore invalidated the symlink from /etc/ceph/ceph.conf -> /etc/pve/ceph.conf

To make it clear, the ceph.conf is stored in the distributed cluster filesystem under /etc/pve/ceph.conf, the one in /etc/ceph/ceph.conf is only symlinked.

Try to create that symlink and test if it solves your issue.

This is as well mentioned in our docs here:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_create_initial_ceph_configuration

 

[Opnek]

It is working now. Thank you very much for your help.

 

[mikefnasr]

Hello,

just ran with the same issue, latest pve and ceph reef
4th node ceph was not initialized, symlink not created, so i created it rebooted and still not initialized

ls -la /etc/ceph/ on newly added node
lrwxrwxrwx 1 root root 18 Apr 25 14:22 ceph.conf -> /etc/pve/ceph.conf
-rw-r--r-- 1 root root 92 May 2 2023 rbdmap

ceph -s on newly added node
cluster:
id: 6c4d3ad6-76d8-4db8-ab55-150dbaea6bea
health: HEALTH_OK

services:
mon: 3 daemons, quorum pve1,pve2,pve3 (age 2w)
mgr: pve2(active, since 3w), standbys: pve1, pve3
mds: 1/1 daemons up, 2 standby
osd: 6 osds: 6 up (since 2w), 6 in (since 3w)

data:
volumes: 1/1 healthy
pools: 8 pools, 225 pgs
objects: 321.56k objects, 1.2 TiB
usage: 3.4 TiB used, 7.4 TiB / 11 TiB avail
pgs: 225 active+clean

io:
client: 444 KiB/s rd, 2.2 MiB/s wr, 19 op/s rd, 140 op/s wr

 

[mikefnasr]

after updating all other PVE nodes, everything is working as expected