[SOLVED] mesh network advice wanted

[RobFantini]

Hello
we have a 7 node ceph cluster which is running fine.

the ceph network uses two 10G switches and Mellanox ConnectX-4 cards. we use bond_mode active-backup.

each node has pci slots to add three more dual port connectx cards.

So I am considering mesh network to eliminate the switches, and gain more speed.
However it seems that a 7 node mesh network will have a complicated /etc/network/interfaces configuration and a lot of wires to keep straightened out.

My question : In our case will a mesh network make things a lot more reliable then using switches?

 

[aaron]

A Mesh for 7 nodes? 6 cables from each node to connect to all the others? I wouldn't do that. This seems to be much too complicated to be handled efficiently and likely to be misconfigured at some point by accident.

If you want better latency I would switch over to a 25G network.

 

[RobFantini]

A Mesh for 7 nodes? 6 cables from each node to connect to all the others? I wouldn't do that. This seems to be much too complicated to be handled efficiently and likely to be misconfigured at some point by accident.

If you want better latency I would switch over to a 25G network.

that was what I thought.

for us 10G is more then enough bandwidth based on zabbix latency graphs. so we'll keep our set up as is..

Thank you.