Mounting CEPHFS on a client

[Rosario Contarino]

I would like to mount CEPHFS on a Client.

Since CEPHFS version is Nautilus, I decided to use, as client, a container running CentOS 7. It might have well been an external physical machine, just happened I wanted to try with a container. Yes, CEPHFS is already installed on ProxMox and working fine as repository for all templates and images of containers and VMs.

Now, CEPH recommend to use ceph-deploy on a node to deploy the necessary ceph-client on the client target (centOS 7).

Thing is ProxMox nodes do not have ceph-deploy installed.

How do you recommend to install the necessary ceph-client packages on CentOS 7 in order to be able to mount CEPHFS?

I would love to see a piece of documentation like this one from RedHat, also for ProxMox 6.x.

Thank you.
Rosario

 

[tom]

Now, CEPH recommend to use ceph-deploy on a node to deploy the necessary ceph-client on the client target (centOS 7).

I do not see any ceph-deploy action on "3. Creating Ceph File System Client Users".

 

[Rosario Contarino]

I do not see any ceph-deploy action on "3. Creating Ceph File System Client Users".

Of course not.

As I wrote, ceph-deploy is referenced only in the official CEPH documentation (see here).

Besides, RedHat has done its job and has created their repository to install all the necessary packages for their CEPHFS clients (Ubuntu, CentOS, etc).

What I am asking you is: Which packages do I need to install on CentOS 7 in order to be able to mount CEPHFS ?

 

[Rosario Contarino]

Of course not.

As I wrote, ceph-deploy is referenced only in the official CEPH documentation (see here).

Besides, RedHat has done its job and has created their repository to install all the necessary packages for their CEPHFS clients (Ubuntu, CentOS, etc).

What I am asking you is: Which packages do I need to install on CentOS 7 in order to be able to mount CEPHFS ?

For the time being, a low-cost solution is:

1. add EPEL repo to CentOS (yum install epel-release), then
2. yum install ceph

 

[tom]

Of course not.

As I wrote, ceph-deploy is referenced only in the official CEPH documentation (see here).

Besides, RedHat has done its job and has created their repository to install all the necessary packages for their CEPHFS clients (Ubuntu, CentOS, etc).

What I am asking you is: Which packages do I need to install on CentOS 7 in order to be able to mount CEPHFS ?

Why should we document something what is already documented by the creator of these OS? I do not get your point.

(Ceph-deploy is not used on Proxmox VE)

 

[Rosario Contarino]

Why should we document something what is already documented by the creator of these OS? I do not get your point.

(Ceph-deploy is not used on Proxmox VE)

Forget the other examples, those were just examples. Let me try to be more clear for you.

Can you please point me out to your documentation where you show which packages need to be installed on a ceph client in order to mount cephfs?

 

[Rosario Contarino]

Why should we document something what is already documented by the creator of these OS? I...

(Ceph-deploy is not used on Proxmox VE)

Because what others document for their OSs start from the assumption that not only the client but also the cluster is built on their OSs.

While what I am trying to do is to mount CEPHFS, built on your OS, onto another client OS.

Example is, you don't use ceph-deploy, which instead is the recommended way from the ceph community to deploy a client.

 

[tom]

Forget the other examples, those were just examples. Let me try to be more clear for you.

Can you please point me out to your documentation where you show which packages need to be installed on a ceph client in order to mount cephfs?

This is not documented as this is not the primarily use case for Ceph on Proxmox VE.

 

[tom]

Example is, you don't use ceph-deploy, which instead is the recommended way from the ceph community to deploy a client.

I do not see any ceph-deploy on Chapter 3.3 on https://access.redhat.com/documenta...file_system_guide/deploying-ceph-file-systems

(Ceph-deploy is used to create ceph and cephfs, but not client access)

 

[Rosario Contarino]

This is not documented as this is not the primarily use case for Ceph on Proxmox VE.

Well, then you should probably alert potential users of the fact that, although you made available CEPHFS, its access from external client is not supported (or, as you say, you didn't want it to be its primary use), rather than stating all over that you now have support to CEPHFS (without clarifying those limitations).

 

[Rosario Contarino]

I do not see any ceph-deploy on Chapter 3.3 on https://access.redhat.com/documenta...file_system_guide/deploying-ceph-file-systems

(Ceph-deploy is used to create ceph and cephfs, but not client access)

I have already answered to your note above and I have never said it was in the RedHat document (I wander from where you interpreted it).

I will re-answer for clarity and for future readers: ceph-deploy is mentioned in the official CEPH documentation. RedHat documentation does not need to mention it because they use a different way to deploy clients.

 

[tom]

I really lost the point in this thread - how can I help?

You have a running CephFS, you have the client howto from other OS.

 

[Rosario Contarino]

Client OS mount fails with:

[root@ct-test-02 /]# mount -t ceph 10.244.70.35:6789:/shared /mnt/ceph -o name=1,secretfile=/etc/ceph/ceph.client.1.secret

failed to load ceph kernel module (1)

mount error 1 = Operation not permitted


Do you have any idea on why?

 

[tom]

failed to load ceph kernel module (1)

you cannot load kernel modules in containers.

 

[Rosario Contarino]

you cannot load kernel modules in containers.

Can the module be loaded into the kernel's host prior to start the LXC container?

 

[tom]

try ceph-fuse client.

 

[Rosario Contarino]

try ceph-fuse client.

Already tried. Fails in loading another module.

[root@ct-test-02 ceph]# ceph-fuse -d -n client.1 /mnt/ceph/

2019-08-16 13:00:38.835 7f2adc4c3e00 0 ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-fuse, pid 12815

2019-08-16 13:00:38.859 7f2adc4c3e00 -1 init, newargv = 0x5635bdf23d50 newargc=7

ceph-fuse[12815]: starting ceph client

fuse: device not found, try 'modprobe fuse' first

ceph-fuse[12815]: fuse failed to start

2019-08-16 13:00:38.899 7f2adc4c3e00 -1 fuse_mount(mountpoint=/mnt/ceph) failed.​

Could it be possible to permanently load those modules into the host kernel? If so, how? I've been looking for those modules but I don't know where they are in PVE

 

[tom]

Already tried. Fails in loading another module.

Did you enable "FUSE" feature in your container configuration?

 

[Rosario Contarino]

Did you enable "FUSE" feature in your container configuration?

I'll let you know on Monday. I only have access to the cluster for few hours a day. In the meanwhile if you have a way to let me load the module on the host kernel, I would much appreciate it.
Best regards
Rosario

 

[Alwin]

Please use the CephFS mount on multiple containers with care, as it may raise conflicts on the host kernel. An alternative approach is to use CephFS as directory storage and use mountpoints on the CT. But as noted in the post below the performance may not be what you expect.
https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/page-6#post-263016