Incorrect file permission set on file `/etc/pve/ceph.conf` causes `ceph` user could not access the ceph config file in PVE cluster

[qiongzhu]

I'm trying to install radosgw + ceph dashboard in an experimental 3-nodes PVE v8.2.2 cluster with Ceph v18.2.2. Here is a problem I found:

Ceph v18.2.2 introduces a new feature 'Multi-Site' in dashboard, which need access additional 3 types of information (`realm` / `zone` / `zonegroup`). However, this information could not be retrieved via standardized S3 API, so the ceph developer choose to invoke `radosgw-admin` cli from dashboard (which means ceph-mgr process), to get the required information.

The `radosgw-admin` cli utility reads config file `/etc/ceph/ceph.conf` by default. But in PVE, the file is a symlink to file `/etc/pve/ceph.conf` as follows

lrwxrwxrwx 1 root root 18 May 18 15:31 ceph.conf -> /etc/pve/ceph.conf

And the file limits its owner/group and permission, make `ceph` user cannot read it:

root@pve8-ceph1:~# ls -l /etc/pve/ceph.conf
-rw-r----- 1 root www-data 682 May 18 15:31 /etc/pve/ceph.conf
root@pve8-ceph1:~# sudo -u ceph cat /etc/pve/ceph.conf
cat: /etc/pve/ceph.conf: Permission denied

ceph-mgr process is started with root privilege, but it drops its privilege shortly, and switched to `ceph:ceph`. When I access dashboard which hosted in this process, `radosgw-admin` invoked with `ceph:ceph` could not work properly.

I'm trying to `chown` or `chmod` on file `/etc/pve/ceph.conf`, but it does not work on PVE cluster filesystem `/etc/pve`

root@pve8-ceph1:/etc/pve# ls -l /etc/pve/ceph.conf
-rw-r----- 1 root www-data 682 May 18 15:31 /etc/pve/ceph.conf
root@pve8-ceph1:/etc/pve# chmod 644 /etc/pve/ceph.conf
chmod: changing permissions of '/etc/pve/ceph.conf': Operation not permitted
root@pve8-ceph1:/etc/pve# chown root:ceph /etc/pve/ceph.conf
chown: changing ownership of '/etc/pve/ceph.conf': Operation not permitted

I can make ceph dashboard works by remove symlink `/etc/ceph/ceph.conf`, and copy the file from `/etc/pve/ceph.conf`. But this method is definitely problematic in future, those config files need be synchronized manually, I think the correct way is set correct owner/group + permission on file `/etc/pve/ceph.conf`

Any advices on this?

 

[geoah]

Hey @qiongzhu, did you ever figure out a way to setup radosgw by any chance?
Seem to be bumping into the same - or similar - issue.

 

[UdoB]

-rw-r----- 1 root www-data 682 May 18 15:31 /etc/pve/ceph.conf

Simple (not necessarily the correct) solution: add the ceph user to the group www-data. adduser ceph www-data

 

[geoah]

@UdoB Woo! Thank you for this, really appreciated. No longer seems to be complaining, so correctness is less important

 

[qiongzhu]

Simple (not necessarily the correct) solution: add the ceph user to the group www-data. adduser ceph www-data

No. This does not work because: in file `/usr/lib/systemd/system/ceph-mgr@.service` , ceph-mgr is launched with parameter `--setuser ceph --setgroup ceph`, which drops any other group permissions.

Here are methods to make ceph happy, you can choose any one of them:
1. the ceph.conf file owner is `ceph`
2. the ceph.conf file group is `ceph`
3. the ceph.conf file permissions allow `others` group to read. eg: 644
4. modify ceph-mgr@.service, which make system upgrade becomes harder

 

[UdoB]

No. This does not work

Well..., it did work for me, in my specific situation. But yes, I also wrote "(not necessarily the correct) solution".

Thank for adding those hints! (I've not cross-checked.)

 

[qiongzhu]

Hey @qiongzhu, did you ever figure out a way to setup radosgw by any chance?
Seem to be bumping into the same - or similar - issue.

Yes. Both radosgw and dashboard can work properly since PVE 8.2. The main problem is ceph.conf file permissions which make 'multi-site' in ceph dashboard complains with 500 error. There are some workarounds above, ugly but useful.

 

[blackline]

Since /etc/pve/ceph.conf can't be chmod'ed or chown'ed, the only option for me was to modify /usr/lib/systemd/system/ceph-mgr@.service:

systemctl stop ceph-mgr@YOURNODE.service
nano /usr/lib/systemd/system/ceph-mgr@.service

# change:
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
# to:
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup www-data
# safe and quit: ctrl + s; ctrl + x
systemctl daemon-reload
systemctl start ceph-mgr@YOURNODE.service

 

[herzkerl]

Since /etc/pve/ceph.conf can't be chmod'ed or chown'ed, the only option for me was to modify /usr/lib/systemd/system/ceph-mgr@.service:

systemctl stop ceph-mgr@YOURNODE.service
nano /usr/lib/systemd/system/ceph-mgr@.service

# change:
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
# to:
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup www-data
# safe and quit: ctrl + s; ctrl + x
systemctl daemon-reload
systemctl start ceph-mgr@YOURNODE.service

Thank you very much—that finally solved it!

 

[penetal]

modify /usr/lib/systemd/system/ceph-mgr@.service:

Another way to do the same edit is using the edit command of systemctl

systemctl edit ceph-mgr@.service

Add the lines (the first one is empty to say "delete old exec line"):

[Service]
ExecStart=
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup www-data

While I do not know if this is better or worse than modifying the original service unit file, at least it uses the "official" and more visible way to modify the service, showing that there is an override file in use:

root@metal01:~# systemctl status ceph-mgr@$(hostname).service 

● ceph-mgr@metal01.service - Ceph cluster manager daemon
     Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/ceph-mgr@.service.d
             └─ceph-after-pve-cluster.conf
             /etc/systemd/system/ceph-mgr@.service.d
             └─override.conf