Mesh network with Infiniband cards for Ceph

[Rudy]

Hello,

I'm currently configuring a three nodes cluster with dual ethernet 1Gbps and dual infiniband cards ""MT25408 ConnectX Mellanox Technologies" (without switch).
I would like to configure the infiniband cards for ceph by using a full mesh network as explained here with broadcast bond but it doesn't seem to work (ping doesn't work). See config below

iface ib0 inet manual
iface ib1 inet manual
auto bond1
iface bond1 inet static
        address 10.0.1.11
        netmask 255.255.255.0
        slaves ib0 ib1
        bond_miimon 100
        bond_mode broadcast
        pre-up modprobe ib_ipoib
        pre-up echo connected > /sys/class/net/ib0/mode
        pre-up echo connected > /sys/class/net/ib1/mode
        mtu 65520

Using this other tuto or by changing bond_mode to e.g. "active_backup", I manage to make 2 nodes communicate (removing the mesh).

Before testing the mesh routing solution, could you please tell if it is possible to make ib interfaces work in multicast broadcast mode. Maybe I'm missing a configuration somewhere e.g. in opensm ...

Thank you in advance for your help or advices,
Kind regards

 

[mir]

infiband only supports active/passive bonding. Eg. both interfaces cannot be active at the same time. You cannot create a 3 node cluster with infiniband without a switch.

 

[wolfgang]

Hi,

you can try a routed configuration
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2

 

[Rudy]

infiband only supports active/passive bonding. Eg. both interfaces cannot be active at the same time. You cannot create a 3 node cluster with infiniband without a switch.

Indeed, broadcast mode didn't probably work because of this.

Hi,

you can try a routed configuration
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2

Yes, I will try a routed configuration. It was my intention.

Thank you both for your reply.
Regards,
Rudy