Cluster and ceph across two public networks

[DKTL]

Hi'

We have a proxmox ceph cluster, where each node, mgr, mon, osd, has two bonded network interfaces, bond0 and bond1, each of which on their own separate network. During installation, only bond0 was engaged on all nodes for ceph network settings public_network and cluster_network, however now is the time to engage bond1 as well, or, if having bond0 and bond1 engaged simultaneously is not possible, then it would be highly desirable to replace bond0 with bond1.

Internet searches on how to engage a second network in a proxmox ceph cluster for public_network and cluster_network are a bit inconclusive, and the task seems a bit risky, if one does not step carefully and does things in the right order. Thus, if some in the community has experiences, then the sharing thereof will be highly appreciated.

Thanks.

Best regards.

Thomas.

 

[shrdlicka]

Hi,

the only thing you can do to use both interfaces would be splitting the public & cluster network [1]. Alternativly I have a "rough" guide what todo if you would like to change interfaces for the "cluster_network".


What I tried is just moving the cluster_network to a different network. This was my /etc/pve/ceph.conf before modifying it:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 192.168.18.81/20
     fsid = bbcf0c63-9c4f-4731-b5f1-cd28ed0a9259
     mon_allow_pool_delete = true
     mon_host = 192.168.18.81 192.168.18.82
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 192.168.18.81/20

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.ticket-cl1]
     host = ticket-cl1
     mds_standby_for_name = pve

[mon.ticket-cl1]
     public_addr = 192.168.18.81

[mon.ticket-cl2]
     public_addr = 192.168.18.82

First step was to read who monitors (mon) need to be handled for network changes [2]. I chose the route of adding a new monitor in the network manually [3]. I added it to the node with IP *.81 my the last command in guide [3] looked like this ceph-mon -i some-mon-name --public-addr 10.10.11.81.

I had three nodes in my cluster and two monitors. It got a bit finickey for me. I change my ceph.conf but disabling (NOT destroying) caused me not getting any info in the webGUI. I'm pretty sure this was a quorum problem. Two mons with the old network and only one with the new caused some issue.

When I had two (of three) mons in the new network it worked

This is the ceph.conf now:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.10.11.81/24
     fsid = bbcf0c63-9c4f-4731-b5f1-cd28ed0a9259
     mon_allow_pool_delete = true
     mon_host = 10.10.11.83 10.10.11.82
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 10.10.11.81/20

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.ticket-cl1]
     host = ticket-cl1
     mds_standby_for_name = pve

[mon.ticket-cl2]
     public_addr = 10.10.11.82

[mon.ticket-cl3]
     public_addr = 10.10.11.83

To have the services automatically started, I recreated them via the GUI. I had to remove an entry from mon_host = 10.10.11.83 10.10.11.82 or else the GUI didn't let me create a new mon host on one node even though it was deleted.

[1] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/
[2] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address
[3] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#adding-a-monitor-manual

 

[DKTL]

Hi'

Ceph documentation https://docs.ceph.com/en/pacific/rados/configuration/network-config-ref/#ceph-networks states as follows.

Ceph functions just fine with a public network only. However, Ceph allows you to establish much more specific criteria, including multiple IP network and subnet masks for your public network.

So it would seem to me that I should be able to use both bond0 and bond1 for the public network, yet I am still missing a real life example to lean against .

Best regards.

Thomas.