Unable to add ceph storage

[breakaway9000]

Hi,

I'm testing ceph in a 3 node cluster. I followed the official guide (https://pve.proxmox.com/wiki/Ceph_Server) and created a ceph cluster with 3 x nodes, each having 4 x Disks attached. My ceph cluster talks to each other using 10.10.10.0/24 network. I have managed to get as far as actually adding the storage to my cluster, but I've gotten stuck.

Here's the OSD config:

I then created a new CEPH pool:

Checked the health status is OK:

Then I went to add the storage like below and clicked "Add". So far so good.

But at this point the actual datastore shows 0.00 kB, and if I click on it in the "Server View" and click on "Content", it says "rbd error: rbd: couldn't connect to the cluster! (500)"

What am I doing wrong?

 

[Ashley]

You missed this bit :

You also need to copy the keyring to a predefined location.

Note that the file name needs to be storage id + .keyring . storage id is the expression after 'rbd:' in /etc/pve/storage.cfg which is my-ceph-storage in the current example.
# cd /etc/pve/priv/
# mkdir ceph
# cp /etc/ceph/ceph.client.admin.keyring ceph/my-ceph-storage.keyring

However I would re-create the pool with more PGS.

 

[breakaway9000]

Doh.. managed to get it mounted now. Can you recommend some reading on the best practices for PGs and best practices for storing virtual machine disks? There is a link to a guide in the proxmox Ceph how-to guide but that doesn't seem to make any sense - it seems to make up PGs for things that are not really relevant for this use case.

 

[Ashley]

Do you plan to add any more OSD's in the near future?

 

[udo]

However I would re-create the pool with more PGS.

Hi,
you don't need to recreate the pool. You can extend the PGs of an pool online (but create IO). But never reduce the PGs!

Udo

 

[Ashley]

Hi,
you don't need to recreate the pool. You can extend the PGs of an pool online (but create IO). But never reduce the PGs!

Udo

Correct, but if it's an empty pool (which this seems to be as only just got added correctly into Proxmox) it is easier/quicker to do.

 

[breakaway9000]

Do you plan to add any more OSD's in the near future?

Yes - I do, this is a test environment after all. How do you expand the number of PGs in in a pool? Does this have to be done through the CLI? Don't see any options to do it in the GUI.

 

[Mike_TBF]

Yes - I do, this is a test environment after all. How do you expand the number of PGs in in a pool? Does this have to be done through the CLI? Don't see any options to do it in the GUI.

Yes, you need to do this on the CLI. See the Ceph docs here for some pointers on how to choose the right number of PGs and the actual commands to do it:
docs.ceph.com/docs/master/rados/operations/placement-groups
(unfortunately I am not allowed to post links, so add the "http" yourself. )

 

[Ashley]

Yes - I do, this is a test environment after all. How do you expand the number of PGs in in a pool? Does this have to be done through the CLI? Don't see any options to do it in the GUI.

If your looking to add more than a minimum of : 1024

ceph osd pool set {pool-name} pg_num 1024

If you have no / minimal data then

ceph osd pool set {pool-name} pgp_num 1024

If you have data in the pool then slowly increase pgp_num after each increase has taken effect.