[SOLVED] rbd error: couldn'd connect to the cluster! (500)

[Nemesis11]

Hi.

We are using Proxmox 4.1 in production with 3 nodes and an NFS storage for images, isos, etc.

We would like to use Proxmox with Ceph Server, so i installed Proxmox with Ceph Server in a Lab to test it.
I'm using 4 identical computers with 1 boot disk and 4 SSDs each for ceph.

Everything went ok, but when i try to view the content of the ceph datastore i receive a "rbd error: couldn'd connect to the cluster! (500)"

In the wiki for version 3 of Proxmox there is this step "You also need to copy the keyring to a predefined location"
http://pve.proxmox.com/wiki/Ceph_Server#Ceph_Client

But i see the keyring "/etc/pve/priv". Yesterday i did what the wiki says:

# cd /etc/pve/priv/
# mkdir ceph
# cp /etc/ceph/ceph.client.admin.keyring ceph/my-ceph-storage.keyring

I did this step and proxmox cluster, after a reboot, stopped working. It did not start the cluster service.

So my question is, is this step valid in Proxmox 4.1?

If yes, why did the cluster service stopped working? If no, what should i do to resolve "rbd error: couldn'd connect to the cluster! (500)"?

My "/etc/pve/storage.cfg"

rbd: ceph
  monhost 10.133.20.1:6789;10.133.20.2:6789;10.133.20.3:6789;10.133.20.4:6789
  pool ceph
  content rootdir,images
  krbd
  username admin

dir: local
  path /var/lib/vz
  content iso,vztmpl,rootdir,images
  maxfiles 0

My "/etc/pve/ceph.conf"

[global]
  auth client required = cephx
  auth cluster required = cephx
  auth service required = cephx
  cluster network = 10.133.20.0/24
  filestore xattr use omap = true
  fsid = 1a8e1665-480b-4e35-9fca-3eb557b8245c
  keyring = /etc/pve/priv/$cluster.$name.keyring
  osd journal size = 5120
  osd pool default min size = 1
  public network = 10.133.20.0/24

[osd]
  keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.0]
  host = prox1
  mon addr = 10.133.20.1:6789

[mon.2]
  host = prox3
  mon addr = 10.133.20.3:6789

[mon.1]
  host = prox2
  mon addr = 10.133.20.2:6789

[mon.3]
  host = prox4
  mon addr = 10.133.20.4:6789

Thanks in advance.

 

[udo]

Hi.

...

# cd /etc/pve/priv/
# mkdir ceph
# cp /etc/ceph/ceph.client.admin.keyring ceph/my-ceph-storage.keyring

Hi,
the name of .keyring must be the same like in storage.cfg.
In your case /etc/pve/priv/ceph/ceph.keyring

My "/etc/pve/storage.cfg"

rbd: ceph
  monhost 10.133.20.1:6789;10.133.20.2:6789;10.133.20.3:6789;10.133.20.4:6789
  pool ceph
  content rootdir,images
  krbd
  username admin

...

You have create pool ceph - not rbd?

ceph osd lspools

I have in monhost simply the IPs:

monhost 172.20.1.61 172.20.1.62 172.20.1.63

Udo

 

[Nemesis11]

Hi,
the name of .keyring must be the same like in storage.cfg.
In your case /etc/pve/priv/ceph/ceph.keyring

Hi.
Doing a "cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ceph.keyring" it worked. Now i can browse the ceph datastore.

Many thanks.

I thought that this only made sense in Proxmox 3 and in Proxmox 4 it was not necessary. The strange thing is that i did this step yesterday and it killed the Proxmox Cluster.
Now it works great.

Thanks one more time