Proxmox CephFS Permission Denied

[rrrrrrr]

We're succesfully using Ceph on Proxmox, and have started to attempt to use CephFS.
We are able to mount, and create a file, but can then not write to the file, it shows the below error:
root@<redacted>:/mnt/ceph# echo "test" > /mnt/ceph/testfile
-bash: echo: write error: Operation not permitted
root@<redacted>:/mnt/ceph# ls
test4 test5 test6 test7 testfile


Ceph Version: 17.2.7
Proxmox Version: 8.1.3

Ceph config steps (with redactions):
# these are the steps i have documented for setting up cephfs mounts

# [ceph cluster] create cephfs
ceph fs volume create docker

# [ceph cluster] setup permissions keyring for authenticating the mount on the client
ceph auth rm client.dockermountuser
ceph auth get-or-create client.dockermountuser mds 'allow rw' osd 'allow class-read object_prefix rbd_children, allow rw tag cephfs pool=cephfs.docker.meta, allow rw tag cephfs pool=cephfs.docker.data' mon 'allow r' > /etc/ceph/ceph.client.dockermountuser.keyring
ceph auth get-key client.dockermountuser > /etc/ceph/ceph.client.dockermountuser.keyring
cat /etc/ceph/ceph.client.dockermountuser.keyring

# [client] configure the mount
mkdir -p /etc/ceph
echo "<redacted>" > /etc/ceph/ceph.client.dockermountuser.keyring
mount -a
add-apt-repository cloud-archive:yoga
apt install ceph-common -y
mkdir /mnt/ceph
# add to fstab
<redacted>:6789,<redacted>:6789,<redacted>:6789:/docker /mnt/ceph ceph name=dockermountuser,secretfile=/etc/ceph/ceph.client.dockermountuser.keyring,noatime,_netdev 0 0

# [client] refresh token and remount (testing auth permissions)
umount /mnt/ceph
echo "<redacted>" > /etc/ceph/ceph.client.dockermountuser.keyring
mount -a

Any ideas on why this could be occurring? We followed all the instructions and have searched across these forums and the Internet for potential solutions to no avail.

 

[hd--]

Hello rrrr,

I have followed your steps so far and have not discovered any obvious errors.
Since the client can already connect to the monitor, I assume that the error is actually in the permissions that you set in the key.
In the documentation [0] they use another tool ceph fs authorize which sets the following permissions:

client.foo
  key: *key*
  caps: [mds] allow r, allow rw path=/bar
  caps: [mon] allow r
  caps: [osd] allow rw tag cephfs data=cephfs_a

So the major difference in privileges is allow class-read object_prefix rbd_children, so i'd recommend reducing the restrictions step by step and check if it works afterwards.

If this does not help, i'd try to connect via the admin keyring and check if this is working.

[0] https://docs.ceph.com/en/latest/cephfs/client-auth/#syntax

 

[rrrrrrr]

Appreciate your fast response!

When attempting to mount with the below:
[client.dockermountuser]
key = <redacted>
caps mds = "allow r fsname=docker, allow rw fsname=docker path=/docker"
caps mon = "allow r fsname=docker"
caps osd = "allow rw tag cephfs data=docker"

We receive: mount error 5 = Input/output error

When using:
[client.dockermountuser]
key = <redacted>
caps mds = "allow r, allow rw path=/docker"
caps mon = "allow r"
caps osd = "allow *"

It works correctly. This is not ideal though as you would expect.

Running ceph -s on the client side shows:
monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
[errno 13] RADOS permission denied (error connecting to the cluster)

But this seemingly is unrelated, as it shows this way regardless of the mount working/not working.

 

[sb-jw]

fsname=docker

Check whether the file system is really called docker and not cephfs ceph fs ls

Running ceph -s on the client side shows:

You also need to specify the keyfile and username.

 

[rrrrrrr]

Check whether the file system is really called docker and not cephfs ceph fs ls


You also need to specify the keyfile and username.

root@<redacted>:~# ceph fs ls
name: swarm, metadata pool: swarm_metadata, data pools: [swarm_data ]
name: docker, metadata pool: cephfs.docker.meta, data pools: [cephfs.docker.data ]
name: testvolume, metadata pool: cephfs.testvolume.meta, data pools: [cephfs.testvolume.data ]

Trying with fsname=cephfs also doesn't work, as we presume is expected considering the above.

Makes sense re: keyfile and username for ceph -s.