Storage Problems With Proxmox/Ceph

[Nicolas Dey]

Thank you for your answer, this helps a lot (it confirms something I was not 100% sure).
I tried to rereun the command that was timing out:

…
# ps aux | grep ceph
root  28597  0.0  0.0 513460 13960 ?  Sl  15:18  0:00 /usr/bin/rados -p test -m 10.2.0.201,10.2.0.202,10.2.0.203 -n client.root --keyring /etc/pve/priv/ceph/cephstorage.keyring --auth_supported cephx df
…

If such a command is already running, another one is timing out as well… but running one at once after having started a clean CEPH instance leaded to the following error that helped me finding the issue:

# /usr/bin/rados -p test -m 10.2.0.201,10.2.0.202,10.2.0.203 -n client.root --keyring /etc/pve/priv/ceph/cephstorage.keyring --auth_supported cephx df
2016-12-12 16:04:17.505486 7f2fc7988a00  0 librados: client.root authentication error (22) Invalid argument
couldn't connect to cluster: (22) Invalid argument
…

I've then tried the following:

# /usr/bin/rados -p test -m 10.2.0.201,10.2.0.202,10.2.0.203 -n client.admin --keyring /etc/pve/priv/ceph/cephstorage.keyring --auth_supported cephx df
pool name  KB  objects  clones  degraded  unfound  rd  rd KB  wr  wr KB
test  0  1  0  0  0  67  52  18  4
  total used  222848  1
  total avail  11680444192
  total space  11680667040
…

The solution was to replace 'username root' by 'username admin' in /etc/pve/storage.cfg'. Now I don't have anymore RBD error, and I am able to create a VM with a disk on the CEPH Storage. After VM creation:

# /usr/bin/rados -p test -m 10.2.0.201,10.2.0.202,10.2.0.203 -n client.admin --keyring /etc/pve/priv/ceph/cephstorage.keyring --auth_supported cephx ls
rbd_directory
rbd_id.vm-1015-disk-1
rbd_header.78da9238e1f29

I was then stuck with a VM not starting. According to dmesg:

…
[272859.852718] Key type ceph registered
[272859.852808] libceph: loaded (mon/osd proto 15/24)
[272859.853470] rbd: loaded (major 250)
[272859.855050] libceph: mon2 10.2.0.203:6789 feature set mismatch, my 106b84a842a42 < server's 40106b84a842a42, missing 400000000000000
[272859.856150] libceph: mon2 10.2.0.203:6789 missing required protocol features
[272869.642691] libceph: mon1 10.2.0.202:6789 feature set mismatch, my 106b84a842a42 < server's 40106b84a842a42, missing 400000000000000
[272869.643808] libceph: mon1 10.2.0.202:6789 missing required protocol features
[272879.627453] libceph: mon0 10.2.0.201:6789 feature set mismatch, my 106b84a842a42 < server's 40106b84a842a42, missing 400000000000000
[272879.628612] libceph: mon0 10.2.0.201:6789 missing required protocol features
…

This is solved (https://access.redhat.com/solutions/2591751) by: 'ceph osd crush tunables hammer'

Thank you a lot, guys, for your help, and I hope my notes will help some others as well, in the future…
Regards,
-- N.D.

 

[spirit]

This is solved (https://access.redhat.com/solutions/2591751) by: 'ceph osd crush tunables hammer'

Thank you a lot, guys, for your help, and I hope my notes will help some others as well, in the future…
Regards,
-- N.D.

this is only when you use krbd. (because kernel rbd is not updated to last jewel version).

if you use librbd (remove krbd option from /etc/pve/storage.cfg), it'll work fine with jewel tunables.

krbd is only needed for containers.

 

[Nicolas Dey]

Yes, you are right: I did what you were suggesting, and did 'ceph osd crush tunables optimal'. And it's still working without error. Thanks!

 

[gkovacs]

pveceph install -version, is only for new install. (and proxmox has not yet updated/supported jewel has a bug existed in old version).

for updating a existing cluster, you need to change your /etc/apt/sources.list.d/ceph.list, and update to jewel.

Read the ceph release note on ceph.com, before updating.

Speaking of Jewel: it supports CephFS. Is there any way currently to mount CephFS as storage in Proxmox or do we have to wait until official support comes?