[SOLVED] CEPH - rbd error: rbd: couldn't connect to cluster (500)

[alchemyx]

Hello,I did CEPH installation according to this - http://pve.proxmox.com/wiki/Ceph_ServerEverything went fine. Done all steps, CEPH seems to work:

root@proxmox-A:/etc/pve/priv/ceph# pveceph  lspoolsName                       size     pg_num                 useddata                          2         64                    0metadata                      2         64                    0rbd                           2         64                    0

storage.cfg:

rbd: rbd	monhost 10.10.10.1,10.10.10.2,10.10.10.3	pool rbd	content images	username admindir: local	path /var/lib/vz	content images,iso,vztmpl,rootdir	maxfiles 0

But when I click on RBD entry I get rbd error: rbd: couldn't connect to cluster (500)Do you have maybe some clues how to solve that? Thanks!

 

[alchemyx]

So one thing was wrong, how I defined monlist (it should be spaces not commas). Also I did change the name to avoid some weird issues but it still does not help:

dir: local	path /var/lib/vz	content images,iso,vztmpl,rootdir	maxfiles 0rbd: shared	monhost 10.10.10.1 10.10.10.2 10.10.10.3	pool rbd	content images	username admin

PS. I have no idea why it formats so weirdly here so pasting it to pastebin also - http://pastebin.com/Ee0S3GD0OK Solved NOW. Most stupid error ever - a typo in key name: /etc/pve/priv/ceph/shred.keyring instead of shared.keyring. Sorry!

 

[MACscr]

Are we sure its supposed to be spaces? The wiki actually has semicolons between them:

# from /etc/pve/storage.cfg
rbd: my-ceph-storage
     monhost 10.10.10.1;10.10.10.2;10.10.10.3
     pool rbd
     content images
     username admin

Though im getting the same connection error as well with spaces or semicolons. =/

 

[MACscr]

Nevermind, Resolved. Issue was the keyring as well. I thought that was only needed if you didnt use the gui. Sad they dont automated that. Really seems half baked.

 

[MACscr]

So others don't waste time, i had to do the following in my storage.cfg for the gui to connect to it:

rbd: ceph-ssd
    monhost 10.10.0.100:6789 10.10.0.104:6789 10.10.0.108:6789
    content images,rootdir
    pool rbd
    krbd
    username admin

EDIT: I lied. I was on the wrong tab. Still get same 500 error as you did originally. So weird.

 

[D.P.]

Hi,
Have you solved this problem? I have the same error after installation according to the http://pve.proxmox.com/wiki/Ceph_Server tutorial.

 

[tomas666]

Hi,
I'm getting same error rbd: couldn't connect to cluster
proxmox 4.2
I've tried almost everything
keyring is presented
from command line everything works
just web gui don't see space available, cant use for VMs
content is unavailable thus rbd: couldn't connect to cluster (500)

 

[tomas666]

and please clarify what is right syntax for monitor host
semicolon? space? comma? with or without ports ?
btw i'm using mesh network for ceph, neverthless network and ceph works

 

[Davyd]

For me, everything is working fine with this storage config, without ports:

rbd: ceph-rbd
        monhost 10.200.0.3;10.200.0.1;10.200.0.2
        username admin
        content images
        pool rbd

When I built this cluster, I have the same problem, but it was my mistake with keyring file location.
Or you can disable ceph auth by setting this lines in ceph.conf

[global]
         auth client required = none
         auth cluster required = none
         auth service required = none

 

[trekkygeek]

copy the keyring into /etc/pve/priv/ceph and rename to the same name as your pool.

Ex.:

storage.cfg:
rbd: ceph-ssd
monhost x.x.x.x:6789;y.y.y.y:6789;z.z.z.z:6789
content rootdir,images
username admin
pool ssd

rbd: ceph-sata
monhost x.x.x.x:6789;y.y.y.y:6789;z.z.z.z:6789
content rootdir,images
username admin
pool sata

#/etc/pve/priv/ceph# ls -l
total 1
-rw------- 1 root www-data 63 Nov 17 23:51 ceph-sata.keyring
-rw------- 1 root www-data 63 Nov 17 23:51 ceph-ssd.keyring

 

[aschmitt]

Quick Question. Where is the actual Key that we have to copy located?

 

[aschmitt]

haha. Never mind, I found it in the Ceph Server documentation. For the record, I'll copy it here for anyone else:

You also need to copy the keyring to a predefined location.
Note that the file name needs to be storage id + .keyring . storage id is the expression after 'rbd:' in /etc/pve/storage.cfg which is my-ceph-storage in the current example.
# cd /etc/pve/priv/
# mkdir ceph
# cp /etc/ceph/ceph.client.admin.keyring ceph/my-ceph-storage.keyring

 

[RealVaVa]

I have same trouble.

After copying keyring to pve/ceph directory on the first node, problem was fixed, but only on the first node.
Example

cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ceph-rbd.keyring

ceph-rbd - it is name of my pool

How I can fix that on second node?

 

[Alwin]

@RealVaVa, you wrote on a thread marked as solved, it is not likely that anyone will answer your question here. For such, open up a new thread.