howto mount a second cephfs of another ceph cluster

[stefws]

We've got two external ceph clusters both working fine with pve5.2 for RBDs.

From 1. cluster we mounted a CephFS using it to store vzdumps,
but we also would like to mount a CephFS from the 2. cluster.

1. cluster is found in /etc/ceph/ceph.conf + ceph.client.admin.keyring
2. cluster is currently found in /etc/ceph/ceph2.conf + ceph2.client.admin.keyring

/etc/fstab contains:
none /var/lib/ceph/backup fuse.ceph ceph.id=admin,_netdev,defaults 0 0

Could we add something like and making this refers 2. cluster?
none /var/lib/ceph/backup2 fuse.ceph ceph.id=admin,_netdev,defaults 0 0

 

[stefws]

figured it out

created a new client entity on 2. cluster named it: client.cfs like this:

# create a new standalone keyring as not to fool w/cluster db
~#ceph-authtool /tmp/keyring -C
# added a new entity
~#ceph-authtool /tmp/keyring -g -n client.cfs
# added needed capabilities for cephfs mounting
~#ceph-authtool /tmp/keyring -n client.cfs --cap mds "allow" --cap mon "allow r" --cap osd "allow rw pool data"
# verified keyring
~#ceph-authtool /tmp/keyring -l
[client.cfs]
key = AQCv...==
caps mds = "allow"
caps mon = "allow r"
caps osd = "allow rw pool data"

# imported into ceph cluster/mon db
~#ceph auth import -i /tmp/keyring
# verifyied from cluster
~#ceph auth get client.cfs
exported keyring for client.cfs
[client.cfs]
key = AQCv...==
caps mds = "allow"
caps mon = "allow r"
caps osd = "allow rw pool data"

# copied keyring to client(s)
~#scp /tmp/keyring clientX:/etc/ceph/ceph.client.cfs.keyring

# added this to client(s)' /etc/fstab
monitor-node:6789:/ /var/lib/ceph/backup2 ceph name=cfs,secretkey=AQCv...==,_netdev,defaults,noatime 0 0

# now I can mount from both clusters
~#mount /var/lib/ceph/backup2
~# df | grep backup
ceph-fuse 949698560 112254976 837443584 12% /var/lib/ceph/backup
10.0.3.1:6789:/ 1610469376 0 1610469376 0% /var/lib/ceph/backup2

Will try to change mount of 1. cluster to not use fuse but kernel ceph module...

Also need to get right capabilities as I can't write properly on the mount...

 

[stefws]

altered caps and copied the client.cfs keyring to /etc/ceph/ceph.client.cfs.keyring and I can write to mount. no sure whether it was the new caps or having the keyring on the clients that solved this, anyway set these caps in cluster and keyring file on clients:

[client.cfs]
key = AQCv...==
caps mds = "allow *"
caps mon = "allow r"
caps osd = "allow *"