Ceph cluster address clarification requested

P

PicklesTheOtt

New Member
Jun 10, 2022
5
0
1
All of the nodes in my HA cluster have two NICs. A 1gb for public network access (192.168.1.x/24) and a 10gb for the cluster link (10.10.50.x/24). When I setup the cluster, it asked me to choose which adapter to choose for the link. Naturally, I selected the 10gb adapter. That works all well and good.

When I setup the ceph, it asked me to specify the public and cluster adapters separately. So I did. Now, in the ceph configuration file, I see where "cluster link" is specified, but the public addresses appear for the monitors. Should I have set both the public and private networks as the 10gb? Am I going to run into performance issues? I don't want the ceph to communicate over the public network.

Here's the ceph configuration from the first node:
Code: 
[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.10.50.131/24
     fsid = 6b5aec33-8838-42c5-91e2-b465a4e3a47a
     mon_allow_pool_delete = true
     mon_host = 192.168.1.131 192.168.1.132 192.168.1.133 192.168.1.134
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 192.168.1.131/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mon.pmha01]
     public_addr = 192.168.1.131

[mon.pmha02]
     public_addr = 192.168.1.132

[mon.pmha03]
     public_addr = 192.168.1.133

[mon.pmha04]
     public_addr = 192.168.1.134
 
shrdlicka said:
Hi

before I write something. I think this post [1] sums it up very well. If not, don't hesitate to ask :).


[1] https://forum.proxmox.com/threads/g...tworks-for-ceph-and-proxmox.58439/post-269388
Click to expand...
Thank you for the quick reply! In that post, there's three links; heartbeat, private, and public. When I setup the ceph, I only specified two networks. Is the heartbeat link configured automatically? Also, is "cluster" and "private" considered the same? If so, then I believe this is setup the way I expected.
 
You can create an extra network for heartbeat signals [1]. If there is a Ceph cluster network uses it for object replication and recovery traffic. The public network would do everything (if there is no cluster network). The post I linked before suggested the faster link for the cluster network :). But you can try to see if you run into problems with the current setup.


[1] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ceph-networks
[2] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#cluster-network
 
cluster network is to have a dedicated network for replication/recovery between osd. (osd->osd)

all other traffic is going to public network (client->mon , mon->mon, osd->mon, client-> osd).

if cluster network is not defined, osd replication/recovery is going to public network.
 
You must log in or register to reply here.
Top Bottom