Proxmox 4.1-15 HA Cluster with CEPH Storage

[Javier Estellés Dasí]

Hi!

I'm trying to setup an HA cluster, using a CEPH for the storage.

I configured everything on the 3 nodes, (CEPH Server and Proxmox 4 HA Cluster) several times on virtual and physical servers without any problems.

Now, I'm testing a new setup with 3 nodes and ssd HD's for journaling, but I have a problem with rbd storage for my virtual machines :-(. I cannot access to content in this storage... always get a awful message saying...

rbd error: rbd: couldn't connect to the cluster! (500)

I followed the steps of the wiki and I have also copied the file /etc/ceph/ceph.client.admin.keyring on /etc/pve/priv/ceph/vms.keyring.

Can anyone give me an idea of what I can do?

Thanks...

 

[wolfgang]

Hi,

can you send the config file
/etc/pve/storage.cfg

 

[Javier Estellés Dasí]

Hi,

can you send the config file
/etc/pve/storage.cfg

dir: local
path /var/lib/vz
content iso,rootdir,images,backup,vztmpl
maxfiles 5

lvm: LVM_1
vgname pve
content images
shared
nodes iesbroch1

lvm: LVM2
vgname pve
content images
shared

rbd: vms
monhost 192.168.1.1,192.168.1.2,192.168.1.3
pool rbd
content images
username admin

Attached configuration process document (IP's 192.168.1.242,192.168.1.2,192.168.241.240)

Output from ceph -s
ceph -s

cluster 23f67d9a-d024-4ae0-820e-1bd756864010
health HEALTH_OK
monmap e3: 3 mons at {0=192.168.1.1:6789/0,1=192.168.1.2:6789/0,2=192.168.1.3:6789/0}
election epoch 28, quorum 0,1,2 0,1,2
osdmap e28: 3 osds: 3 up, 3 in
pgmap v250: 128 pgs, 2 pools, 0 bytes data, 0 objects
103 MB used, 5586 GB / 5586 GB avail
128 active+clean

 

[wolfgang]

rbd: vms
monhost 192.168.1.1,192.168.1.2,192.168.1.3
pool rbd
content images
username admin

correct would be

rbd: vms
monhost 192.168.1.1;192.168.1.2;192.168.1.3
pool rbd
content images
username admin

 

[Javier Estellés Dasí]

Thanks you! Rapid reply

I changed the configuration file and has not solved the problem. It does not seem that this is the reason for the error.

In fact, I have a installation replica for testing and works with both comma and semicolon. ¿?

dir: local
path /var/lib/vz
content vztmpl,images,iso,rootdir
maxfiles 0

rbd: vms
monhost 10.2.0.108,10.2.0.109,10.2.0.110
content images
username admin
pool rbd

rbd: vms2
monhost 10.2.0.108;10.2.0.109;10.2.0.110
content images
username admin
pool rbd

Anyway, from this moment, I write thi with semicolon ;-).

Thanks

PS: New configuration document (attached)

 

[Javier Estellés Dasí]

[SOLVED]

I had made a pveceph purge... and reconconfigure ceph cluster and client... and now is working correctly

Thanks