CEPH - Initial config IP Public/Cluster

[Ma907xb]

Hello,

Running Virtual Environment 6.1-5 . I currently have a 4 node setup. I'm connecting my fourth node to the other 3 nodes. I joined the server to the cluster, then installed ceph. After installing CEPH my browser was refreshed and the initial installation window was closed.

How do I set the IP address of my ceph public/cluster networks? I have 2 separate networks.

How do i see what my other nodes have set for there IPs?

The other 3 nodes have been operating on a public/cluster network previous to me joining this fourth server.

The 4th is 192.168.50.40. Do i just need to add that IP to the /etc/pve/ceph.conf as below?

[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 192.168.51.10/24
fsid = ***
mon_allow_pool_delete = true
mon_host = 192.168.50.10 192.168.50.20 192.168.50.30 192.168.50.40
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 192.168.50.10/24

Thank you

 

[t.lamprecht]

How do I set the IP address of my ceph public/cluster networks? I have 2 separate networks.

How do i see what my other nodes have set for there IPs?

The ceph cluster network is set cluster wide in Proxmox VE through the /etc/pve/ceph.conf file all nodes share.

The 4th is 192.168.50.40. Do i just need to add that IP to the /etc/pve/ceph.conf as below?

Is a monitor running on it (a even count of monitors is not recommended), as only then it would make sense to add that IP.

Your config shows that your ceph setup uses 192.168.51.10/24 (== 192.168.51.0/24) as cluster network.
The public network is currently on the same CIDR. If you want to change one of them just alter those, for example for moving the public network to 192.168.51.0/24 you would do:

public_network = 192.168.51.0/24

And restart all services + OSDs (not on each change this has to be done, but to be sure)

 

[Ma907xb]

Is a monitor running on it (a even count of monitors is not recommended), as only then it would make sense to add that IP.

why is an even number of monitors not recommended? Is this the same for managers?

 

[t.lamprecht]

why is an even number of monitors not recommended?

Monitor use a the PAXOS algorithm to form a quorum. In any quorum you need at list more than half of the votes in the cluster to have a majority.
For three monitors quorum is 2
For four monitors quorum is 3

So it isn't inherently bad, but causes more cluster (quorum) traffic and monitors require some CPUs. That means you use more resources while gaining nothing - 3 monitors is enough and recommended for cluster between, roughly, 3 and 8 nodes, over that I'd maybe go with 5 monitors, but that depends on the setup, performance and expected load.

Is this the same for managers?

They do not have quorum mechanisms themself, but in smaller clusters it's not really required to have 4 managers.

So you can use that "many", but you will require slightly more resource usage without having a benefit. If that's fine for you, then no need to change that, just more of a FYI

 

[Ma907xb]

Thank you! I appreciate the FYI. I went and added the additional 4th node without adding monitors or managers. Once the OSD is added. CEPH will start to recplicate onto the new OSDs? I just need to wait for this process to finish?

 

[t.lamprecht]

Once the OSD is added. CEPH will start to recplicate onto the new OSDs?

Yes, will automatically rebalance data to new active (started and marked as "in") OSDs.

From the screenshot It seems that you need to restart some OSDs, one after the other, due to a software update of Ceph.