Cannot connect to ceph after install

[cclloyd]

Fresh single node install currently, with the intent to add 2 more nodes later that will use the Ceph storage from the first node.


I installed the first node and installed ceph.

I have 2 virtual bridges
- vmbr0, 10.0.1.2/16 - general network
- vmbr3, 10.10.10.2/24 - ceph network


I configured ceph with the network 10.10.10.2/24 for the public/cluster network, added 2 SSDs as OSDs, then created a pool and added it as storage.

But after I do all that, I try to create a VM using ceph as the storage, and I get an error:

> TASK ERROR: unable to create VM 100 - error with cfs lock 'storage-CEPH1': rbd error: got lock timeout - aborting command

What am I doing wrong?

ceph.cfg

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.10.10.2/24
     fsid = 7b8de1ea-92ae-4244-82df-2118e12963fc
     mon_allow_pool_delete = true
     mon_host =  10.10.10.2
     osd_pool_default_min_size = 2
     osd_pool_default_size = 2
     public_network = 10.10.10.2/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

Network settings for node:
https://i.imgur.com/TTU9cFi.png

 

[Jarek]

You are propably going to use ceph pool with failure domain = host and min_size = 2, so you neeed at least 2 hosts.
Now you have a choices (from good to very bad):
- add second node;
- create rule with failure domain = osd;
- change min_size of the pool to 1.

 

[Alwin]

Fresh single node install currently, with the intent to add 2 more nodes later that will use the Ceph storage from the first node.

Ceph needs 3x nodes minimum. And it will need OSDs on all of them. Anything else is just careless, if you need your data.