Cannot open ceph.conf

[klowet]

Hello

I'm running a Proxmox and a separate Ceph cluster for RBD image storage. In Proxmox, I get this error, eg. when generating backups.

2019-09-04 13:45:49.559767 7f8d85b850c0 -1 did not load config file, using default settings.
2019-09-04 13:45:49.565157 7f8d85b850c0 -1 Errors while parsing config file!
2019-09-04 13:45:49.565159 7f8d85b850c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2019-09-04 13:45:49.565160 7f8d85b850c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2019-09-04 13:45:49.565160 7f8d85b850c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2019-09-04 13:45:49.565990 7f8d85b850c0 -1 Errors while parsing config file!
2019-09-04 13:45:49.565994 7f8d85b850c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2019-09-04 13:45:49.565994 7f8d85b850c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2019-09-04 13:45:49.565995 7f8d85b850c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory

That error is correct. /etc/ceph/ceph.conf does not exist because Ceph lives on a separate cluster.

# pveversion --verbose
proxmox-ve: 6.0-2 (running kernel: 5.0.15-1-pve)
pve-manager: 6.0-5 (running version: 6.0-5/f8a710d7)
pve-kernel-5.0: 6.0-5
pve-kernel-helper: 6.0-5
pve-kernel-4.15: 5.4-6
pve-kernel-5.0.15-1-pve: 5.0.15-1
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.12-pve1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-3
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-5
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-6
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1[CODE]

Thank you

 

[tim]

Please share your storage configuration

 

[klowet]

# cat /etc/pve/storage.cfg
dir: local
    disable
    path /var/lib/vz
    content backup,vztmpl,iso
    maxfiles 3
    shared 0

zfspool: local-zfs
    disable
    pool rpool/data
    content rootdir,images
    sparse 1

rbd: rbd
    content images
    krbd 0
    monhost ips-of-mons-here
    pool rbd
    username mycephusername

The rest of the file are some nfs shares.

 

[Craig St George]

same as me external ceph and seeing this on snapshot rollback backups and clones

 

[MarcelW]

I tried to restore a VM from external CIFS Storage to a seperate Ceph Storage and got the same errors.

 

[sg90]

Just create the empty file : /etc/ceph/ceph.conf

You don't need to put anything in it, but you may want to add some ceph client values that the VM's will pickup (cache e.t.c)

 

[Craig St George]

thanks

 

[Elie.SF]

Just create the empty file : /etc/ceph/ceph.conf

You don't need to put anything in it, but you may want to add some ceph client values that the VM's will pickup (cache e.t.c)

Is there really no risks that this empty file will conflicts with another/future CEPH configuration ?

 

[sg90]

Is there really no risks that this empty file will conflicts with another/future CEPH configuration ?

If your CEPH cluster is separate to Proxmox then the ceph.conf file is only every read by the QEMU RBD process for any client specfic entries.

Will have 0 impact on any separate CEPH cluster.

 

[klowet]

Is there really no risks that this empty file will conflicts with another/future CEPH configuration ?

I've created an empty /etc/ceph/ceph.conf file on this cluster. Works fine.

 

[dragoangel]

Actually to have properly working ceph dashboard with Object Gateway adding ceph to www-data doesn't help.

-rw-r----- 1 root www-data 1.8K Sep 10 12:24 /etc/ceph/ceph.conf

Next error is appears with enabled debug logs:

[dashboard DEBUG root] Error 1 executing ['radosgw-admin', '-c', '/etc/ceph/ceph.conf', '-k', '/var/lib/ceph/mgr/ceph-node/keyring', '-n', 'mgr.node', 'sync', 'status']: global_init: unable to open config file from search list /etc/ceph/ceph.conf

To enabled debug:

ceph dashboard debug enable
ceph config set mgr mgr/dashboard/log_level debug

To get working Object Gateway user or group of /etc/ceph/ceph.conf must be set to ceph or file permissions must be set to 644.

 

[Lauch42]

I added an override to the ceph-mgr systemd definition that copies the ceph.conf prior to service start with correct permissions for ceph.
Cleanest solution I could come up with.

systemctl edit ceph-mgr@.service

[Service]
Environment=CEPH_CONF=/var/lib/ceph/mgr/ceph-%i/ceph.conf
ExecStartPre=+/usr/bin/install --owner=ceph --mode=400 -T /etc/ceph/ceph.conf /var/lib/ceph/mgr/ceph-%i/ceph.conf