Enquiry on setting ceph for 2 nodes

[uglyxiaodi]

Dear All,

I have bought 2 servers and my host has given me 2 ip segment 192.168.0.35/29and 192.168.0.139/29
I have installed ceph on my first node and create a monitor and below is the configuration ceph file however I have encountered an issue in creating monitor for node 2
( with a network: 192.168.0.139/29)

No active IP found for the requested ceph public network '192.168.0.35/29' on node 'chicago03' (500)
Is the above error cause by firewall for both my nodes?
I have tried to edit /etc/ceph/ceph.conf and set the public network to 192.168.0.0/29 and cluster network to 10.10.10.0/29 however the error is "No active IP found for the requested ceph public network '192.168.0.0/29' on node 'chicago03' (500)"

[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 10.10.10.35/29
fsid = ac9e9642-532f-4d87-bb1f-0c64e2f52bb9
mon_allow_pool_delete = true
mon_host = 192.168.0.35
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 192.168.0.35/29

[client]
keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.chicago01]
public_addr = 192.168.0.35

Many thanx in advance

 

[aaron]

Don't!

Ceph expects a network segment to share between the nodes. The network needs to be fast (10GBit or more) with low latency. It also needs at least 3 nodes to work properly and keep your data secure.
Additionally, you shouldn't run the Ceph traffic over an interface that is already used for other stuff because Ceph can and will use up a lot of bandwidth.
The docs have a section on the preconditions.

What might be useful in your situation is to use ZFS on both nodes for the VM storage and use ZFS replication to have a copy on the other node in case of a disaster.

 

[uglyxiaodi]

Dear aaron,

Currently, Im doing the testing phase where I will begin to start with 2 nodes, eventually there will be more than 3 nodes,

Can I clarify network segment to share between the nodes ? which means there must be an internal network which these nodes share?
I cant use the public ipv4 address of both nodes to link up?

Thank you

 

[aaron]

If you use the public IPs, they will be in different network segments. The PVE tooling and to a certain degree the Ceph tooling will have a problem with that as they assume that the Ceph networks (Ceph public and the optional Ceph Cluster) will be in one network segment.

Additionally, there are other concerns than just configurational ones. How fast is that link between the nodes and how is the latency?

Ceph needs a fast network (10Gbit or more) and low latency. It can also very quickly use up all the bandwidth in a 10Gbit network. This means, you should use a dedicated network for it, so it does not interfere with other traffic.

If you plan to create a cluster, there are other concerns regarding the network as well. The PVE cluster communication is using the corosync protocoll. It does not need a lot of bandwidth, but really needs low latency. Thats why it is recommended to use at least one dedicated physical network just for Corosync. You can configure multiple corosync links and it will switch by itself, should one link become unavailable due to high latency or a broken connection.

If you plan to use the PVE HA stack, a stable Corosync connection becomes mandatory! Without HA in use, a somewhat flaky corosync connection will mean that you won't be able to do certain tasks if there is a problem. Tasks like changing some configuration or starting a VM. If you use HA, Corosync is used to determine if a node is still part of the cluster. If a node loses the Corosync connection to the quorate part of the cluster (majority), it will fence itself (hard reset) after about 2 minutes in order to make sure that the HA guests on it are off. Now, if you have Corosync running on the same network with another service that can use up all the bandwidth (like Ceph), you run the risk, that all nodes lose the connection via Corosync due to high latency, caused by the other services using up all the bandwidth. If that situation stays like that long enough (~2min), you will see your whole cluster do a hard reset.

One last word regarding testing a Ceph cluster: Do start with 3 nodes. That is the minimum amount of nodes needed to run a healthy Ceph cluster. If you run a 2 node cluster, Ceph will complain that it is not able to create a third replica. And do not set the PG size to 2 and min_size to 1. That is a recipe for data loss/corruption!

I hope that this explanation helps to understand why the recommendations are the way they are