CephFS installation

[MewBie]

Since there isn't a lot of info on how to get CephFS running on Proxmox I thought it was a nice idea to write how I got a proof of concept style setup "working" (there are some limitations that I currently don't know how to fix).

First up this is my setup:
hostname: pm1
Running on a physical server as a single node Proxmox cluster and a single node Ceph cluster. This computer is also a host for all VM's used.

hostname: prox1, prox2, prox3
Three VM's to have access to a working Proxmox and Ceph cluster.

hostname: ubuntu1
Ubuntu Live VM.

On prox1-3 you follow the standard Ceph server install guides. Once you get the cluster up and running it's time to start working on the MDS.

First you need to install ceph-mds on one of the nodes, so run on one of them:

apt install ceph-mds

Next you can follow the Ceph manual:
http://docs.ceph.com/docs/master/install/manual-deployment/#adding-mds
Note that clustername isn't the same as your Proxmox clustername, in step 5 the clustername is used to load the corresponding .conf file. So if you use 'blabla' as clustername it will try to load blabla.conf. If it fails it will load default settings from ceph.conf if I understand correctly. After mine failed in step 5 I just did step 6 and everything seems to be working.

Don't try to check the Ceph status now, it won't show up yet.

Up next is creating the pools and the actual fs, follow this Ceph manual for that:
http://docs.ceph.com/docs/master/cephfs/createfs/

And finally run this command, change the pool names if you used different ones:

ceph osd pool application enable cephfs_metadata cephfs

Congrats, you now have a running cephfs on Proxmox.
To test if it's working I used Ubuntu 16.04 LTS Live in a VM, open a terminal and use the following command to install needed Ceph components.

sudo apt install ceph-common ceph-fuse

With that installed you can follow this Ceph manual:
http://docs.ceph.com/docs/master/cephfs/fuse/
In step 2 you need to run:

sudo mkdir -p /etc/pve/priv
sudo scp {user}@{server-machine}:/etc/pve/priv/ceph.client.admin.keyring /etc/pve/priv/ceph.client.admin.keyring

In my setup the CephFS mount has only write access for root, but it works and files placed in the folder are still there when I reboot the Ubuntu VM and install the client again. write access can be solved by creating a folder as root and setting the permissions on the folder.

 

[TheDevouringOne]

OMG I have been working on this for like 2 weeks. I am trying this now. Thank you so much!

I wish it was easier to get CephFS working

edit:
ceph -s still showing mds 0/0/1

Ill try to see if I can get it to mount

 

[MewBie]

It should show mds 1/1/1 before it works. Did you create the fs before you checked the status?

I also updated the first post a bit to say which manual to use. Still no links though.

 

[TheDevouringOne]

I didn't check until after the CephFS creation =/. I am currently now trying to get fuse to work on a VM. So far from what I can tell it says the services and everything are running. Hopefully tonight I can mess with it more and get something to mount.

EDIT 1:
FUSE is getting connection timeout (110)
demsg:
libceph: bad option at 'secretfile=/etc/pve/priv/admin.secret

I had put a secret file there on the client and in the node with the mds

 

[MewBie]

I just simply copied the ceph.client.admin.keyring from the servers /etc/pve/priv folder to the clients /etc/pve/priv folder in step 2 of the ceph-fuse manual and it worked.

 

[TheDevouringOne]

Started with just doing that and I have been messing with it since.

The previous timeout 110 error was because the ceph versions were different. I have fixed that fired up a ubuntu vm and squared away the ceph versions.

I feel like I have screwed up cephx in the MDS manual setup somehow and I can't figure out where.





Ceph report on my main Node:

 "fsmap": {

        "epoch": 2,

        "compat": {

            "compat": {},

            "ro_compat": {},

            "incompat": {

                "feature_1": "base v0.20",

                "feature_2": "client writeable ranges",

                "feature_3": "default file layouts on dirs",

                "feature_4": "dir inode in separate object",

                "feature_5": "mds uses versioned encoding",

                "feature_6": "dirfrag is stored in omap",

                "feature_8": "file layout v2"

            }

        },

        "feature_flags": {

            "enable_multiple": false,

            "ever_enabled_multiple": false

        },

        "standbys": [],

        "filesystems": [

            {

                "mdsmap": {

                    "epoch": 0,

                    "flags": 12,

                    "ever_allowed_features": 0,

                    "explicitly_allowed_features": 0,

                    "created": "2017-08-20 22:54:22.873706",

                    "modified": "2017-08-20 22:54:22.873707",

                    "tableserver": 0,

                    "root": 0,

                    "session_timeout": 60,

                    "session_autoclose": 300,

                    "max_file_size": 1099511627776,

                    "last_failure": 0,

                    "last_failure_osd_epoch": 0,

                    "compat": {

                        "compat": {},

                        "ro_compat": {},

                        "incompat": {

                            "feature_1": "base v0.20",

                            "feature_2": "client writeable ranges",

                            "feature_3": "default file layouts on dirs",

                            "feature_4": "dir inode in separate object",

                            "feature_5": "mds uses versioned encoding",

                            "feature_6": "dirfrag is stored in omap",

                            "feature_8": "file layout v2"

                        }

                    },

                    "max_mds": 1,

                    "in": [],

                    "up": {},

                    "failed": [],

                    "damaged": [],

                    "stopped": [],

                    "info": {},

                    "data_pools": [

                        1

                    ],

                    "metadata_pool": 2,

                    "enabled": true,

                    "fs_name": "Ceph",

                    "balancer": "",

                    "standby_count_wanted": 0

                },

                "id": 1

            }

        ]

    },

    "mdsmap_first_committed": 1,

    "mdsmap_last_committed": 2,

    "auth": {

        "first_committed": 1,

        "last_committed": 63,

        "num_secrets": 18

    },

Spoiler: ceph.conf

[global]
auth client required = cephx
auth cluster required = cephx
auth service required = cephx
cluster network = 10.10.10.0/24
fsid = xxxxxxxxxxxxxxxxxx
keyring = /etc/pve/priv/$cluster.$name.keyring
mon allow pool delete = true
osd journal size = 5120
osd pool default min size = 2
osd pool default size = 3
public network = 10.10.10.0/24

[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.purple1]
host = purple1

[mon.purple2]
host = purple2
mon addr = 10.10.10.xx:6789

[mon.purple3]
host = purple3
mon addr = 10.10.10.xx:6789

[mon.purple1]
host = purple1
mon addr = 10.10.10.xx:6789

 

[MewBie]

I don't know a whole lot about this, but comparing it to my report it looks like your mds is not running.

"max_mds": 1,
                    "in": [
                        0
                    ],
                    "up": {
                        "mds_0": 4389
                    },
                    "failed": [],
                    "damaged": [],
                    "stopped": [],
                    "info": {
                        "gid_4389": {
                            "gid": 4389,
                            "name": "prox1",
                            "rank": 0,
                            "incarnation": 4,
                            "state": "up:active",
                            "state_seq": 85,
                            "addr": "192.168.1.101:6813/2358132398",
                            "laggy_since": "2017-08-22 22:10:52.415626",
                            "standby_for_rank": -1,
                            "standby_for_fscid": -1,
                            "standby_for_name": "",
                            "standby_replay": false,
                            "export_targets": [],
                            "features": 2305244844532236283
                        }
                    },

 

[TheDevouringOne]

I am banging my head against the wall.

 

[TheDevouringOne]

What does your ceps.conf look like? sorry about blowing you up with questions I appreciate all the help


EDIT:
No idea what any of this means but I ran this: "ceph-mds -i purple1 -d" and its now showing it as running


2017-08-22 17:24:11.820384 7faa7b8ff6c0 0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 19871

starting mds.purple1 at -

2017-08-22 17:24:11.824902 7faa7b8ff6c0 0 pidfile_write: ignore empty --pid-file

2017-08-22 17:24:15.906370 7faa746a7700 1 mds.purple1 handle_mds_map standby

2017-08-22 17:24:15.918804 7faa746a7700 1 mds.0.4 handle_mds_map i am now mds.0.4

2017-08-22 17:24:15.918813 7faa746a7700 1 mds.0.4 handle_mds_map state change up:boot --> up:creating

2017-08-22 17:24:15.919098 7faa746a7700 0 mds.0.cache creating system inode with ino:0x1

2017-08-22 17:24:15.919293 7faa746a7700 0 mds.0.cache creating system inode with ino:0x100

2017-08-22 17:24:15.919304 7faa746a7700 0 mds.0.cache creating system inode with ino:0x600

2017-08-22 17:24:15.919482 7faa746a7700 0 mds.0.cache creating system inode with ino:0x601

2017-08-22 17:24:15.919598 7faa746a7700 0 mds.0.cache creating system inode with ino:0x602

2017-08-22 17:24:15.919728 7faa746a7700 0 mds.0.cache creating system inode with ino:0x603

2017-08-22 17:24:15.919835 7faa746a7700 0 mds.0.cache creating system inode with ino:0x604

2017-08-22 17:24:15.919939 7faa746a7700 0 mds.0.cache creating system inode with ino:0x605

2017-08-22 17:24:15.920077 7faa746a7700 0 mds.0.cache creating system inode with ino:0x606

2017-08-22 17:24:15.920236 7faa746a7700 0 mds.0.cache creating system inode with ino:0x607

2017-08-22 17:24:15.920344 7faa746a7700 0 mds.0.cache creating system inode with ino:0x608

2017-08-22 17:24:15.920485 7faa746a7700 0 mds.0.cache creating system inode with ino:0x609

2017-08-22 17:24:15.956087 7faa6e69b700 1 mds.0.4 creating_done

2017-08-22 17:24:16.926174 7faa746a7700 1 mds.0.4 handle_mds_map i am now mds.0.4

2017-08-22 17:24:16.926181 7faa746a7700 1 mds.0.4 handle_mds_map state change up:creating --> up:active

2017-08-22 17:24:16.926191 7faa746a7700 1 mds.0.4 recovery_done -- successful recovery!

2017-08-22 17:24:16.926217 7faa746a7700 1 mds.0.4 active_start

 

[MewBie]

I can't access the .conf right now but I'm testing CephFS on a stock Proxmox 5 install with Ceph Luminous. So 3 nodes and Ceph installed according to the Proxmox documentation. So besides the mds there are no modifications to ceph.conf.

And the mds code in ceph.conf is something like:

[mds.prox1]
host=prox1

As mentioned in the Ceph manual.

 

[TheDevouringOne]

right below that it mentions

Then make sure you do not have a keyring set in ceph.conf in the global section; move it to the client section; or add a keyring setting specific to this mds daemon. And verify that you see the same key in the mds data directory and ceph auth get mds.{id} output.

so I dropped a key in [MDS]

I am going to disable cephx and see if that will help. I went ahead and got some 10gig gear to connect everything properly. so ill be rebuilding the whole thing from scratch for the 10th ish time lol

 

[MewBie]

I'm running the VM's in bridge mode on my single 1Gb network without problems. When I get home I'll send you the links to the manuals I've used. Maybe you're using a different one.