Ceph Client issue

[Peter Sutherland]

I have installed proxmox 4.4 and clustered 4 servers. 3 of the servers I have installed ceph jewel, and setup a ceph cluster. I created a "servers" pool and shared the storage to all 4 servers. I can get access to the shared ceph "servers" pool storage from only the servers where I have installed ceph. However, on the 4th server where I did not install ceph, but only shared the storage, the storage is not accessible. I then went ahead and installed ceph on the 4th server, but did not configure, then the share "servers" pool storage was accessible.

Is this normal? I had previously setup proxmox 3.4 with ceph, in a similar setup, and did not need to install ceph on non ceph servers. The only difference was this was a proxmox 4.4 install. Has anyone else run into the same issue?

 

[wolfgang]

Hi,

If you install ceph jewel you have to update the librbd min version 0.94 on the client nodes.

The default PVE 4.4 librbd is 0.80 and this is only compatible up to Hammer.

 

[Peter Sutherland]

Thanks Wolfgang,

Would librbd be in the proxmox repository or the ceph repository? I thought I had ran all updates on the client node, before I installed ceph.

 

[wolfgang]

In PVE 4 you have to take the ceph repository to get the current librbd packages.

In PVE 5 it will in our repository.

 

[Peter Sutherland]

Wolfgang, does this mean that I need to install ceph on a non ceph client node? Or do I just add the ceph repository and run normal updates?

 

[wolfgang]

Add the repository and the key and then dist-upgrade and you got everything you need.

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

echo deb https://download.ceph.com/debian-jewel/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list

 

[Peter Sutherland]

thanks wolfgang. Now just to make it a little bit more complex. I have another proxmox 5 node cluster, 3 nodes have ceph jewel installed and configured and 2 of the nodes do not have installed but can access ceph storage. It is on proxmox 4.4 however, on the non ceph nodes librbd is on version 0.80, yet can still connect to ceph shares. If librvd needs to be on version 0.94 then I am not sure how this is working?

 

[wolfgang]

What is the problem?
You have a cluster wide storage config and there is the mon's set.
If the 2 non ceph nodes have connection to the mon and osd (ceph network) you can use them as storage out of the box.

 

[Peter Sutherland]

I get that, but would it still work even though the 2 non ceph nodes are on librbd version 0.80. Just am a little confuse on why on one cluster it seems to not work until I updated librbd and another cluster it seems to work on version 0.80.

 

[wolfgang]

Ok Now I get you,

This is a protocol version problem.

librbd supports the successor and the predecessor.