Installing Ceph MDS on Proxmox

[sdutremble]

I have been trying to install a Ceph MDS on Proxmox and am having difficulties.

First: I am aware that mds is not yet supported on proxmox.
Second: I have seen in the forum that it is likely possible.
Third: I want to look into it.

So I currently have configured a cluster with 3 Proxmox 3.3 with Ceph firefly nodes. All latest updates done on the no-subscription apt.
The Proxmox LAN is 1GB 192.168.255.0/24 and the storage LAN is a 2 X GB NICs bonded together using round-robin on a separate switch at 10.10.10.0/24.
The Proxmox/ceph nodes each have 1 X SSD for the system and 4 x 1TB WD HDD Red drives for OSDs.
I have connectivity everywhere and the Ceph health is OK.
So far, I have 2 KVM using RBD with acceptable performance.

I installed ceph-deploy on all three nodes in order to install a MDS:

apt-get install ceph-deploy

Then, I tried to install one MDS:

ceph-deploy mds create pve1

I got:

root@pve1:/etc/pve/priv/ceph# ceph-deploy mds create pve1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.17): /usr/bin/ceph-deploy mds create pve1
[ceph_deploy][ERROR ] ConfigError: Cannot load config: [Errno 2] No such file or directory: 'ceph.conf'; has `ceph-deploy new` been run in this directory?

I changed directory to where the ceph.conf is on Proxmox:

cd /etc/pve

and now I am getting:

root@pve1:/etc/pve# ceph-deploy mds create pve1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.17): /usr/bin/ceph-deploy mds create pve1
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts pve1:pve1
[ceph_deploy][ERROR ] RuntimeError: bootstrap-mds keyring not found; run 'gatherkeys'

Running gatherkeys, I then get:

root@pve1:/etc/pve# ceph-deploy gatherkeys pve1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.17): /usr/bin/ceph-deploy gatherkeys pve1
[ceph_deploy.gatherkeys][DEBUG ] Checking pve1 for /etc/ceph/ceph.client.admin.keyring
[pve1][DEBUG ] connected to host: pve1 
[pve1][DEBUG ] detect platform information from remote host
[pve1][DEBUG ] detect machine type
[pve1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from pve1.
[ceph_deploy.gatherkeys][DEBUG ] Checking pve1 for /var/lib/ceph/mon/ceph-{hostname}/keyring
[pve1][DEBUG ] connected to host: pve1 
[pve1][DEBUG ] detect platform information from remote host
[pve1][DEBUG ] detect machine type
[pve1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/mon/ceph-{hostname}/keyring on ['pve1']
[ceph_deploy.gatherkeys][DEBUG ] Checking pve1 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[pve1][DEBUG ] connected to host: pve1 
[pve1][DEBUG ] detect platform information from remote host
[pve1][DEBUG ] detect machine type
[pve1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from pve1.
[ceph_deploy.gatherkeys][DEBUG ] Checking pve1 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[pve1][DEBUG ] connected to host: pve1 
[pve1][DEBUG ] detect platform information from remote host
[pve1][DEBUG ] detect machine type
[pve1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from pve1.
root@pve3:/etc/pve# ceph-deploy mds create pve1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.17): /usr/bin/ceph-deploy mds create pve1
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts pve1:pve1
[pve1][DEBUG ] connected to host: pve1 
[pve1][DEBUG ] detect platform information from remote host
[pve1][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: debian 7.6 wheezy
[ceph_deploy.mds][DEBUG ] remote host will use sysvinit
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to pve1
[pve1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.mds][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to create 1 MDSs

The link from /etc/ceph/ceph.conf -> /etc/pve/ceph.conf works but appears to cause some issue.

Can anyone shed some light on what I may be doing wrong?

This should not be this hard.

Please advise.

Serge.

 

[sdutremble]

Adding details to my own post.

I followed this: http://www.sebastien-han.fr/blog/2013/05/13/deploy-a-ceph-mds-server/

Creating things manually and still got an error saying ceph-mds was not found.

I then did:

apt-get install ceph-mds

which installed needed packages and then

root@pve1:/etc/pve# service ceph start mds.0

and the mds.0 service started.

So I am going to try mounting it.

I will report what happens.

Serge

 

[sdutremble]

Mounting with ceph-fuse works. However, you have to install ceph-fuse:

apt-get install ceph-fuse

I mounted the CephFS in a directory (/tmp/CephFS) within each node using the storage LAN and configured the storage as a shared directory within the Proxmox WebGUI.

The new CephFS storage is now available for each host.

I initiated copying files within it and found that it slowed the VMs quite a bit... Quite a bit of IO-Wait. However it is still workable while Ceph is getting stabilized.

I hope it will be much better once the Ceph Health status is OK.

I have not been able to use the kernel mount. I believe ceph support is not baked in the kernel and I can not find the ceph kernel module.

Serge