OpenFabric 3 node cluster only 2 working fabrics - 1 always 'not ok'

[hcorneliussen]

Hello all,

I am currently struggling with setting up a openfabric network across 3 nodes.
One fabric is always showing not ok and will not get up but it is not always the same one.

So the fabric between 1 and 2 is working and 3 is not getting up.
If I now restart node 2 for example the fabric on 3 gets up but 2 stays down and not ok and so forth. So I always only get the connection working between 2 nodes.

Short description of my current hardware:

3 nodes with each 2 network cards for my VMLan.
2 switches at the top that are connected with each other.

Node1:
card 1 -> switch 1
card 2 -> switch 2
Bond2 - active backup both cards
Bridge0 - bond2 (VLAN vnet and VXLan vnet)

Node2:
Node2 - card 1 -> switch 1
Node2 - card 2 -> switch 2
Bond2 - active backup both cards
Bridge0 - bond2 (VLAN vnet and VXLan vnet
)

Node3:
Node3 - card 1 -> switch 1
Node3 - card 2 -> switch 2
Bond2 - active backup both cards
Bridge0 - bond2 (VLAN vnet and VXLan vnet)

On each node I made an active backup bond and connected that bond to a bridge so i can use that bridge in an sdn VLAN vnet and that is working.
If I now define an IP adress on each bridge I can ping across all nodes and MTU is also working fine with jumbo frames without fragmentation.
A vxlan vnet with each node as a peer is also working perfectly.
Now the issue is, that my cluster will grow much more (7 more nodes are on the way) and to manually only setup the peers in a one line input can fast become a bit confusing...
So I wanted to setup a openfabric to streamline the configuration and make it more easy to follow but that leads to the above mentioned issue...
I tried using the bond, I tried using the bridge and I tried using the cards directly but I always get the same issue or nothing working.

I am currently lost and would really appreciate help.

 

[ggoller]

Hi,
OpenFabric only supports point-to-point links, so using a switch between nodes is not supported. I recommend using OSPF and setting ip ospf network point-to-multipoint or using IS-IS and removing the isis network point-to-point (which will force it into broadcast mode). This is not yet supported in the UI, so you would need to add this to the frr.conf.local file.

 

[hcorneliussen]

Hi,

thank you very much for that answer! I seem to misunderstood OpenFabric then in total...
Would you mind telling me if it would be best to use the bond, the bridge or the network cards directly inside the Node Configuration inside the OSPF?

 

[ggoller]

Hi!
OpenFabric is meant as a subset of IS-IS which is optimized for spine-leaf topologies, which inherently use point-to-point links only. It should work on both the bond and directly on the network cards, but I'd recommend activating OSPF on all the physical network cards, as that is just simpler.

 

[hcorneliussen]

Hello Gabriel,

Thank you very much! Will read a bit more up on that matter, test it and get back to you!