Incorrect file permission set on file `/etc/pve/ceph.conf` causes `ceph` user could not access the ceph config file in PVE cluster

[qiongzhu]

I'm trying to install radosgw + ceph dashboard in an experimental 3-nodes PVE v8.2.2 cluster with Ceph v18.2.2. Here is a problem I found:

Ceph v18.2.2 introduces a new feature 'Multi-Site' in dashboard, which need access additional 3 types of information (`realm` / `zone` / `zonegroup`). However, this information could not be retrieved via standardized S3 API, so the ceph developer choose to invoke `radosgw-admin` cli from dashboard (which means ceph-mgr process), to get the required information.

The `radosgw-admin` cli utility reads config file `/etc/ceph/ceph.conf` by default. But in PVE, the file is a symlink to file `/etc/pve/ceph.conf` as follows

lrwxrwxrwx 1 root root 18 May 18 15:31 ceph.conf -> /etc/pve/ceph.conf

And the file limits its owner/group and permission, make `ceph` user cannot read it:

root@pve8-ceph1:~# ls -l /etc/pve/ceph.conf
-rw-r----- 1 root www-data 682 May 18 15:31 /etc/pve/ceph.conf
root@pve8-ceph1:~# sudo -u ceph cat /etc/pve/ceph.conf
cat: /etc/pve/ceph.conf: Permission denied

ceph-mgr process is started with root privilege, but it drops its privilege shortly, and switched to `ceph:ceph`. When I access dashboard which hosted in this process, `radosgw-admin` invoked with `ceph:ceph` could not work properly.

I'm trying to `chown` or `chmod` on file `/etc/pve/ceph.conf`, but it does not work on PVE cluster filesystem `/etc/pve`

root@pve8-ceph1:/etc/pve# ls -l /etc/pve/ceph.conf
-rw-r----- 1 root www-data 682 May 18 15:31 /etc/pve/ceph.conf
root@pve8-ceph1:/etc/pve# chmod 644 /etc/pve/ceph.conf
chmod: changing permissions of '/etc/pve/ceph.conf': Operation not permitted
root@pve8-ceph1:/etc/pve# chown root:ceph /etc/pve/ceph.conf
chown: changing ownership of '/etc/pve/ceph.conf': Operation not permitted

I can make ceph dashboard works by remove symlink `/etc/ceph/ceph.conf`, and copy the file from `/etc/pve/ceph.conf`. But this method is definitely problematic in future, those config files need be synchronized manually, I think the correct way is set correct owner/group + permission on file `/etc/pve/ceph.conf`

Any advices on this?

 

[geoah]

Hey @qiongzhu, did you ever figure out a way to setup radosgw by any chance?
Seem to be bumping into the same - or similar - issue.

 

[UdoB]

-rw-r----- 1 root www-data 682 May 18 15:31 /etc/pve/ceph.conf

Simple (not necessarily the correct) solution: add the ceph user to the group www-data. adduser ceph www-data

 

[geoah]

@UdoB Woo! Thank you for this, really appreciated. No longer seems to be complaining, so correctness is less important

 

[qiongzhu]

Simple (not necessarily the correct) solution: add the ceph user to the group www-data. adduser ceph www-data

No. This does not work because: in file `/usr/lib/systemd/system/ceph-mgr@.service` , ceph-mgr is launched with parameter `--setuser ceph --setgroup ceph`, which drops any other group permissions.

Here are methods to make ceph happy, you can choose any one of them:
1. the ceph.conf file owner is `ceph`
2. the ceph.conf file group is `ceph`
3. the ceph.conf file permissions allow `others` group to read. eg: 644
4. modify ceph-mgr@.service, which make system upgrade becomes harder

 

[UdoB]

No. This does not work

Well..., it did work for me, in my specific situation. But yes, I also wrote "(not necessarily the correct) solution".

Thank for adding those hints! (I've not cross-checked.)

 

[qiongzhu]

Hey @qiongzhu, did you ever figure out a way to setup radosgw by any chance?
Seem to be bumping into the same - or similar - issue.

Yes. Both radosgw and dashboard can work properly since PVE 8.2. The main problem is ceph.conf file permissions which make 'multi-site' in ceph dashboard complains with 500 error. There are some workarounds above, ugly but useful.

 

[blackline]

Since /etc/pve/ceph.conf can't be chmod'ed or chown'ed, the only option for me was to modify /usr/lib/systemd/system/ceph-mgr@.service:

systemctl stop ceph-mgr@YOURNODE.service
nano /usr/lib/systemd/system/ceph-mgr@.service

# change:
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
# to:
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup www-data
# safe and quit: ctrl + s; ctrl + x
systemctl daemon-reload
systemctl start ceph-mgr@YOURNODE.service

 

[herzkerl]

Since /etc/pve/ceph.conf can't be chmod'ed or chown'ed, the only option for me was to modify /usr/lib/systemd/system/ceph-mgr@.service:

systemctl stop ceph-mgr@YOURNODE.service
nano /usr/lib/systemd/system/ceph-mgr@.service

# change:
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
# to:
ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup www-data
# safe and quit: ctrl + s; ctrl + x
systemctl daemon-reload
systemctl start ceph-mgr@YOURNODE.service

Thank you very much—that finally solved it!