rbd error: couldn'd connect to the cluster! (500)

[pdan]

Hello forum,

one more question... installed ceph using pveceph tools. My cluster has very unbalanced osds and only one monitor but I built it up only for testing purpose.

All osds up/in, has HEALTH_WARN condition (maybe due to osd/capacity unbalancing between nodes) and:

root@c10:/etc/pve/priv# ceph status
    cluster bf89586c-4748-40a6-a93e-5b9f0a3bc363
     health HEALTH_WARN 1 pgs stuck unclean
     monmap e3: 1 mons at {1=10.10.0.86:6789/0}, election epoch 5, quorum 0 1
     osdmap e18: 4 osds: 4 up, 4 in
      pgmap v145: 192 pgs, 3 pools, 8 bytes data, 1 objects
            141 MB used, 4001 GB / 4001 GB avail
                   1 active+remapped
                 191 active+clean
root@c10:~# rbd create --size 10 pippo
root@c10:~# rbd ls
pippo
root@c10:~# rbd rm pippo
Removing image: 100% complete...done.
root@c10:~# rbd create --size 10000 pippo
root@c10:~# rbd rm pippo
Removing image: 100% complete...done.

cluster.conf and storage.cfg

root@c10:~# cat /etc/pve/cluster.conf
<?xml version="1.0"?>
<cluster name="casilab" config_version="3">

  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>

  <clusternodes>
  <clusternode name="c10" votes="1" nodeid="1"/>
  <clusternode name="c11" votes="1" nodeid="2"/><clusternode name="c12" votes="1" nodeid="3"/></clusternodes>

</cluster>
root@c10:~# cat /etc/pve/storage.cfg 
dir: local
        path /var/lib/vz
        content images,iso,vztmpl,rootdir
        maxfiles 0

nfs: nfsiso
        path /mnt/pve/nfsiso
        server 10.10.0.129
        export /mnt/share0/dataset1
        options vers=3
        content images,iso
        maxfiles 0

rbd: myrbd
        monhost 10.10.0.86
        pool rbd
        content images
        username admin

root@c10:~#

and finally ceph.conf

root@c10:~# cat /etc/pve/ceph.conf 
[global]
         auth client required = cephx
         auth cluster required = cephx
         auth service required = cephx
         auth supported = cephx
         cluster network = 192.168.0.0/22
         filestore xattr use omap = true
         fsid = bf89586c-4748-40a6-a93e-5b9f0a3bc363
         keyring = /etc/pve/priv/$cluster.$name.keyring
         osd journal size = 5120
         osd pool default min size = 1
         public network = 10.10.0.0/22

[osd]
         keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.1]
         host = c11
         mon addr = 10.10.0.86:6789

pveversion

root@c10:~# pveversion -v
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
pve-manager: 3.2-30 (running version: 3.2-30/1d095287)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-29-pve: 2.6.32-126
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-14
qemu-server: 3.1-34
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-22
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-5
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
root@c10:~#

Any idea?

Pasquale

 

[wahmed]

In your storage.cfg, it should be like this :
rbd: myrbd
monhost 10.10.0.86:6789
pool rbd
........

You are missing the port number in monhost.

 

[pdan]

In your storage.cfg, it should be like this :
rbd: myrbd
monhost 10.10.0.86:6789
pool rbd
........

You are missing the port number in monhost.

OK port 6789 added. Now the 500 error doesn't show in GUI but when I try to create a VM and create storage on rbd pool, I obtain "TASK ERROR: create failed - rbd error: rbd: couldn't connect to the cluster!" and the VM creation aborts.

Pasquale

 

[tom]

keyring problem? (You also need to copy the keyring to a predefined location.)

see http://pve.proxmox.com/wiki/Ceph_Server#Ceph_Pools

 

[pdan]

[SOLVED] Re: rbd error: couldn'd connect to the cluster! (500)

keyring problem? (You also need to copy the keyring to a predefined location.)

see http://pve.proxmox.com/wiki/Ceph_Server#Ceph_Pools

Infact I miss keyring copying in priv/ceph. Now it's all right!

Pasquale