[SOLVED] CephFS Mount Connection Timed Out

[Mihai]

This is on a 3 node cluster.

The versions:

Proxmox Kernel Version
Linux 4.13.13-1-pve #1 SMP PVE 4.13.13-31
Ceph: 12.2.2

Successfully created a cephfs as far as I can tell.
Ccephx is disabled. Though I did create the cephfs while the cluster still had cephx enabled, and I disabled it shortly after.

Cephfs: cephfs-1/1/1 up {0=cephfs=up:active}

I'm using commands:

mount -t ceph 10.10.0.14:6789,10.10.0.13:6789,10.10.0.16:6789:/ /mnt/cephfs

Also tried one monitor only:

mount -t ceph 10.10.0.16:6789:/ /mnt/cephfs

Also tried mounting with fstab.

The fuse-client gives me no feedback when mounting, seems stuck at starting ceph client? Just in case I was missing something, I did start the ceph-fuse client on two hosts and tried to create a file in the directory, but it did not show up on the other host, so I figure it's not working either.


What I see in /var/log/syslog is this:

libceph: mon2 10.10.0.14:6789 socket closed (con state CONNECTING)
libceph: mon1 10.10.0.13:6789 socket closed (con state CONNECTING)
libceph: mon0 10.10.0.16:6789 socket error on write

Anyone have any ideas on what I'm doing wrong?

 

[Alwin]

Did you create all the needed pools?

It is not recommended to disable cephx, as now anyone with access to the ceph network has access to your data.

 

[Mihai]

Yes, I have created a cepfs_data and cepfs_metadata pool and a ceph filesystem on top of that.

 

[MewBie]

For the fuse client you need to copy the ceph.conf file to the clients and put it in the correct folder.

 

[Alwin]

When Ceph is installed on Proxmox, the ceph.conf is already in place. Is cephx really deactivated? Did you also try with the mount wrapper (mount.ceph)? http://docs.ceph.com/docs/luminous/cephfs/kernel/

If you want to try through fuse: http://docs.ceph.com/docs/luminous/cephfs/fstab/

 

[MewBie]

From what I've read using kernel mounting for a CephFS on a server that is running the Ceph services is problematic at best. So probably best to avoid that.
Mounting CephFS using fuse on the same server shouldn't be a problem and in that case ceph.conf is already in place.
Mounting CephFS using fuse on a separate client, this includes VM's, I needed to copy the ceph.conf from Proxmox to the client for it to work.

 

[Mihai]

Ahhh made such a dumb mistake. I was pointing to the Proxmox network, not ceph network address!!!!

Freaking embarrassing.

 

[Mihai]

From what I've read using kernel mounting for a CephFS on a server that is running the Ceph services is problematic at best. So probably best to avoid that.
Mounting CephFS using fuse on the same server shouldn't be a problem and in that case ceph.conf is already in place.
Mounting CephFS using fuse on a separate client, this includes VM's, I needed to copy the ceph.conf from Proxmox to the client for it to work.

You are correct, I just ran into that problem.

Proxmox will FAIL to boot if you try and mount cephfs through fstab on the same server running the services.

I suspect it has to do with the fact that ceph tries to mount prior to ceph services running.

I got around that by mounting the cephfs through a script including a 60 second delay and triggered by an init file at boot.

Though thinking about it now, I really do not need to mount this on the hosts, so to avoid any further problems I should just disable it.