Fabric Mesh more than 3 nodes.

[ErkDog]

I currently have 3 nodes A B C

There are dedicated links for A -> B -> C -> Back to A.

Fabric Mesh is configured and working.

Can I add a 4th node A -> B -> C -> D -> Back to A?

or nTH node?

I'm assuming so, because otherwise you'd have to have a trillion interfaces to go above 3 nodes, but I wanted to check before I go down a rabbit hole and possibly break something.

Thanks,
Matt

 

[aaron]

Can I add a 4th node A -> B -> C -> D -> Back to A?

That is a ring and will introduce other issues, especially as it grows. One is latency and bandwidth concerns as you introduce more hops to get from one node to the other. You also increase chances that the network will break into segments if more than one connection is down.

Ideally, you will always have point to point connections between the nodes. But the number of interfaces needed will grow exponentially with each node. And making sure that each node is connected to each other will also get a lot more complicated. For your own sanity, it is probably better to opt for the regular bond connected to two stacked switches (so that a switch can be down/rebooted)

 

[ErkDog]

that makes sense Aaron, thank you very much. However, that introduces a new (and scary) problem, of migrating the existing 3 nodes to that connection instead of the fabric mesh >.<

I'm not -super- familiar with bonding in Linux. But I'd be using Mikrotik switches most likely, so I'll poke that group about how to configure the Mikrotiks for a Debian Linux bond across two cross connected switches.

However, that leaves me with having to figure out how to change both the Cluster Synchronization & Ceph infrastructure to that new connection.

Any thoughts?

Thanks,
Matt

 

[aaron]

Yeah, Ideally you can test the network setup with other hardware first and go through the different failure scenarios before you apply it to production.

Ideally, it can handle the following things:

• one switch being down
• node 1 losing connection to switch A, while node 2 lost the connection to switch B -> node 1 and 2 can still communicate via the connection between the switches

How that needs to be configured on the switch side, is something you need to find out. It is usually called MLAG (multi chassis link aggregation)