Unable to add ceph storage

breakaway9000

Renowned Member
Dec 20, 2015
91
22
73
Hi,

I'm testing ceph in a 3 node cluster. I followed the official guide (https://pve.proxmox.com/wiki/Ceph_Server) and created a ceph cluster with 3 x nodes, each having 4 x Disks attached. My ceph cluster talks to each other using 10.10.10.0/24 network. I have managed to get as far as actually adding the storage to my cluster, but I've gotten stuck.

Here's the OSD config:
vk2scO.jpg


I then created a new CEPH pool:

q4EbRS.jpg


Checked the health status is OK:
DYp2Sc.jpg


Then I went to add the storage like below and clicked "Add". So far so good.
bw23mu.jpg


But at this point the actual datastore shows 0.00 kB, and if I click on it in the "Server View" and click on "Content", it says "rbd error: rbd: couldn't connect to the cluster! (500)"

What am I doing wrong?
 
You missed this bit :

You also need to copy the keyring to a predefined location.

Note that the file name needs to be storage id + .keyring . storage id is the expression after 'rbd:' in /etc/pve/storage.cfg which is my-ceph-storage in the current example.
# cd /etc/pve/priv/
# mkdir ceph
# cp /etc/ceph/ceph.client.admin.keyring ceph/my-ceph-storage.keyring

However I would re-create the pool with more PGS.
 
Doh.. managed to get it mounted now. Can you recommend some reading on the best practices for PGs and best practices for storing virtual machine disks? There is a link to a guide in the proxmox Ceph how-to guide but that doesn't seem to make any sense - it seems to make up PGs for things that are not really relevant for this use case.
 
Hi,
you don't need to recreate the pool. You can extend the PGs of an pool online (but create IO). But never reduce the PGs!

Udo

Correct, but if it's an empty pool (which this seems to be as only just got added correctly into Proxmox) it is easier/quicker to do.
 
Do you plan to add any more OSD's in the near future?

Yes - I do, this is a test environment after all. How do you expand the number of PGs in in a pool? Does this have to be done through the CLI? Don't see any options to do it in the GUI.
 
Yes - I do, this is a test environment after all. How do you expand the number of PGs in in a pool? Does this have to be done through the CLI? Don't see any options to do it in the GUI.

Yes, you need to do this on the CLI. See the Ceph docs here for some pointers on how to choose the right number of PGs and the actual commands to do it:
docs.ceph.com/docs/master/rados/operations/placement-groups
(unfortunately I am not allowed to post links, so add the "http" yourself. :) )
 
Yes - I do, this is a test environment after all. How do you expand the number of PGs in in a pool? Does this have to be done through the CLI? Don't see any options to do it in the GUI.
If your looking to add more than a minimum of : 1024

ceph osd pool set {pool-name} pg_num 1024

If you have no / minimal data then

ceph osd pool set {pool-name} pgp_num 1024

If you have data in the pool then slowly increase pgp_num after each increase has taken effect.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!