RBD Storage Troubleshooting

[pashadee]

Hi there,

I have configured a 3 node ceph cluster, and setup a pool caled pve_data (as per proxmox wiki on this topic).

I have copied the ceph.client.admin.keyring /etc/pve/priv/ceph/storage_id.keyring

I added the storage in the GUI with the ips of the 3 monitors such as 192.168.1.31:6789;XX:6789;XX:6789

correct pool name (checked over and over), and admin for username ...

It lets me add it with no errors.. shows up in the GUI, but when I click on the Contents tab I just get communication failure (0) .. no other errors...

I'm not sure how to further troubleshoot this, is there log files or something I can watch so I can figure out where and what exactly is failing? I can ping the ceph hosts, ssh in to them, etc etc... there is no iptables running on them... please help!

Thanks in advance guys!

storage.cfg

rbd: ceph_storage
monhost 192.168.1.31:6789;192.168.1.32:6789;192.168.1.33:6789
pool pve_data
content images
username admin

ceph node:

pool data id 0
nothing is going on


pool metadata id 1
nothing is going on


pool rbd id 2
nothing is going on


pool pve_data id 3
nothing is going on

Not sure what else can be useful. (ceph -w)

cluster c675bac9-5531-422f-a561-e724bac8fde2
health HEALTH_OK
monmap e1: 3 mons at {ceph1=192.168.1.31:6789/0,ceph2=192.168.1.32:6789/0,ceph3=192.168.1.33:6789/0}, election epoch 6, quorum 0,1,2 ceph1,ceph2,ceph3
osdmap e18: 3 osds: 3 up, 3 in
pgmap v61: 448 pgs, 4 pools, 0 bytes data, 0 objects
124 MB used, 5571 GB / 5571 GB avail
448 active+clean

 

[pashadee]

I should mention, it's Proxmox 2.3 and yes my admin key is name ceph_storage.keyring, I shouldn't have used the sample. I apologize.

 

[pashadee]

rbd map -m 192.168.1.31 --pool pave_data pve_data_rbd --id admin --keyring /etc/pve/priv/ceph/ceph_storage.keyring
2014-05-14 10:53:27.765576 7f8777b56760 -1 did not load config file, using default settings.
rbd: /sys/bus/rbd/add does not exist!
Did you run 'modprobe rbd' or is your rbd module too old?
rbd: add failed: (2) No such file or directory
root@lenard:/# modprobe rbd
FATAL: Module rbd not found.

 

[spirit]

rbd map -m 192.168.1.31 --pool pave_data pve_data_rbd --id admin --keyring /etc/pve/priv/ceph/ceph_storage.keyring
2014-05-14 10:53:27.765576 7f8777b56760 -1 did not load config file, using default settings.
rbd: /sys/bus/rbd/add does not exist!
Did you run 'modprobe rbd' or is your rbd module too old?
rbd: add failed: (2) No such file or directory
root@lenard:/# modprobe rbd
FATAL: Module rbd not found.

Hi, rbd kernel module is not used by proxmox. (qemu directly connect through librbd).


I see that is proxmox 2.3, maybe librbd is too old for your ceph storage ?

maybe can you try to add ceph wheezy repository, and update last version of librbd package ?

 

[pashadee]

Hi, rbd kernel module is not used by proxmox. (qemu directly connect through librbd).


I see that is proxmox 2.3, maybe librbd is too old for your ceph storage ?

maybe can you try to add ceph wheezy repository, and update last version of librbd package ?

You are the man! thanks so much for that suggestion, here's the solution if anyone else runs in to it. For the record my ceph cluster is firefly release.

On Proxmox Hosts (2.3)

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | apt-key add -

add: deb http://ceph.com/debian/ squeeze main
to: /etc/apt/sources.list.d/ceph.list

apt-get update
apt-get dist-upgrade (this will update more than just your libvirt and rados libraries, so make sure that's what you want to do).

After those steps my ceph storage showed up with proper pool, vailable and used space, etc.

Thanks again!