CEPH: Can we run 2x Cluster network subnet simultanously ?

[DC-CA1]

hi in case of a NIC or switch failure , is i possible to define 2 network ?
( we use bonding no worry , but in case also of LACP issues )

example
cluster_network = 192.168.1.0/24 192.168.2.0/24
public_network = 192.168.4.0/24 192.168.3.0/24

other question , per my understanding Monitor will only listen to 1 adress at a time right ? so it will not help in this case but for cluster ?

 

[gurubert]

The OSDs will also only listen to one IP address.

It is possible to configure multiple IP networks for public and cluster network. This is for the case when your cluster nodes are in separate buildings and routing has to be used between them, not for failover.

 

[DC-CA1]

That is our goal. Mutlti location .

Can you share a config example as i didn't found any

 

[DC-CA1]

So osd is 1 network only ? Or we can define multiple too and it will reconnect onthe second provided ip automatically?

 

[gurubert]

So osd is 1 network only ?

Yes, OSDs bind to one IP only. This IP and the TCP port are stored in the OSD map, part of the cluster map.

 

[mfed]

I used the configuration with multiple cluster and public networks when I needed to migrate my network to new IP subnets. My understanding is that those two settings act like ACL by allowing connections only with the source IP within the specified range. And I believe OSD/MON select the first interface from this range to listen on. I don't think it will do any failover for you.

If you need the failover between interfaces/switches you can create a configuration with a /32 IP address on a dummy interface and the ECMP routing configuration to the physical interfaces, and have your ceph using those IP addresses. It's a more complex configuration, but works beautifully

 

[Drummond Korras]

I used the configuration with multiple cluster and public networks when I needed to migrate my network to new IP subnets. My understanding is that those two settings act like ACL by allowing connections only with the source IP within the specified range. And I believe OSD/MON select the first interface from this range to listen on. I don't think it will do any failover for you.

If you need the failover between interfaces/switches you can create a configuration with a /32 IP address on a dummy interface and the ECMP routing configuration to the physical interfaces, and have your ceph using those IP addresses. It's a more complex configuration, but works beautifully

Hi mfed, could you possibly give an example of the ECMP config please
TIA

 

[Drummond Korras]

I managed to find this guide, I am going to try it out but it looks amazing
https://www-pivert-org.translate.go...l=auto&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp