[SOLVED] CEPH - rbd error: rbd: couldn't connect to cluster (500)

alchemyx

New Member
Apr 2, 2014
7
0
1
Hello,I did CEPH installation according to this - http://pve.proxmox.com/wiki/Ceph_ServerEverything went fine. Done all steps, CEPH seems to work:
Code:
root@proxmox-A:/etc/pve/priv/ceph# pveceph  lspoolsName                       size     pg_num                 useddata                          2         64                    0metadata                      2         64                    0rbd                           2         64                    0
storage.cfg:
Code:
rbd: rbd	monhost 10.10.10.1,10.10.10.2,10.10.10.3	pool rbd	content images	username admindir: local	path /var/lib/vz	content images,iso,vztmpl,rootdir	maxfiles 0
But when I click on RBD entry I get rbd error: rbd: couldn't connect to cluster (500)Do you have maybe some clues how to solve that? Thanks!
 
Last edited:
So one thing was wrong, how I defined monlist (it should be spaces not commas). Also I did change the name to avoid some weird issues but it still does not help:
Code:
dir: local	path /var/lib/vz	content images,iso,vztmpl,rootdir	maxfiles 0rbd: shared	monhost 10.10.10.1 10.10.10.2 10.10.10.3	pool rbd	content images	username admin
PS. I have no idea why it formats so weirdly here so pasting it to pastebin also - http://pastebin.com/Ee0S3GD0OK Solved NOW. Most stupid error ever - a typo in key name: /etc/pve/priv/ceph/shred.keyring instead of shared.keyring. Sorry!
 
Last edited:
Are we sure its supposed to be spaces? The wiki actually has semicolons between them:

Code:
# from /etc/pve/storage.cfg
rbd: my-ceph-storage
     monhost 10.10.10.1;10.10.10.2;10.10.10.3
     pool rbd
     content images
     username admin

Though im getting the same connection error as well with spaces or semicolons. =/
 
Nevermind, Resolved. Issue was the keyring as well. I thought that was only needed if you didnt use the gui. Sad they dont automated that. Really seems half baked.
 
So others don't waste time, i had to do the following in my storage.cfg for the gui to connect to it:

Code:
rbd: ceph-ssd
    monhost 10.10.0.100:6789 10.10.0.104:6789 10.10.0.108:6789
    content images,rootdir
    pool rbd
    krbd
    username admin

EDIT: I lied. I was on the wrong tab. Still get same 500 error as you did originally. So weird.
 
Last edited:
Hi,
I'm getting same error rbd: couldn't connect to cluster
proxmox 4.2
I've tried almost everything
keyring is presented
from command line everything works
just web gui don't see space available, cant use for VMs
content is unavailable thus rbd: couldn't connect to cluster (500)
 
and please clarify what is right syntax for monitor host
semicolon? space? comma? with or without ports ?
btw i'm using mesh network for ceph, neverthless network and ceph works
 
For me, everything is working fine with this storage config, without ports:
Code:
rbd: ceph-rbd
        monhost 10.200.0.3;10.200.0.1;10.200.0.2
        username admin
        content images
        pool rbd
When I built this cluster, I have the same problem, but it was my mistake with keyring file location.
Or you can disable ceph auth by setting this lines in ceph.conf
Code:
[global]
         auth client required = none
         auth cluster required = none
         auth service required = none
 
copy the keyring into /etc/pve/priv/ceph and rename to the same name as your pool.

Ex.:

storage.cfg:
rbd: ceph-ssd
monhost x.x.x.x:6789;y.y.y.y:6789;z.z.z.z:6789
content rootdir,images
username admin
pool ssd

rbd: ceph-sata
monhost x.x.x.x:6789;y.y.y.y:6789;z.z.z.z:6789
content rootdir,images
username admin
pool sata

#/etc/pve/priv/ceph# ls -l
total 1
-rw------- 1 root www-data 63 Nov 17 23:51 ceph-sata.keyring
-rw------- 1 root www-data 63 Nov 17 23:51 ceph-ssd.keyring
 
haha. Never mind, I found it in the Ceph Server documentation. For the record, I'll copy it here for anyone else:

You also need to copy the keyring to a predefined location.
Note that the file name needs to be storage id + .keyring . storage id is the expression after 'rbd:' in /etc/pve/storage.cfg which is my-ceph-storage in the current example.
# cd /etc/pve/priv/
# mkdir ceph
# cp /etc/ceph/ceph.client.admin.keyring ceph/my-ceph-storage.keyring
 
I have same trouble.

After copying keyring to pve/ceph directory on the first node, problem was fixed, but only on the first node.
Example

cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ceph-rbd.keyring

ceph-rbd - it is name of my pool

How I can fix that on second node?
 
@RealVaVa, you wrote on a thread marked as solved, it is not likely that anyone will answer your question here. For such, open up a new thread.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!