Infiniband interface bonding

[plokko]

I'm configuring a 3 node HA cluster with Ceph storage;
this cluster is composed of 3 identical nodes, each node has
2x 1gb ethernet cards
2x 40gb infiniband cards (mellanox-4)

The infiniband cards are connected to a 32 port infiniband switch that is used both for SAN and for public access (via a 10gb sfp+ cable adapters to LAN routers).

I will use two network: a public ip 192.168.100.x and a SAN (Ceph) networking 10.0.10.x

I want to use the two infiniband cards to work both as SAN (Ceph networking) and as public interface

I already configured the infiniband cards to work as networking card.

As redundancy is very important i want to use the infinibad networking as the main network and the ethernet cards as a fallback if a cable is disconnected or the ib switch is down.

I wanted to create something like a "raid 10" for networking:
I tried creating two 802.3ad bond, one for ethernet (bond0) and one for infiniband (bond1) and creating a third bonding (bond2) for active-backup but it won't work.

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface ib0 inet manual

iface ib1 inet manual

iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad
#Ethernet bond

iface bond1 inet manual
    bond-slaves ib0 ib1
    bond-miimon 100
    bond-mode 802.3ad
#IB bond

auto bond2
iface bond2 inet manual
    bond-slaves bond0 bond1
    bond-miimon 100
    bond-mode active-backup
    BONDING_OPTS="mode=1 miimon=100 primary=bond1"
#Main bond

auto vmbr0
iface vmbr0 inet static
    address  192.168.100.211
    netmask  255.255.255.0
    gateway  192.168.100.240
    bridge-ports bond2
    bridge-stp off
    bridge-fd 0
#public interface

Is this solution feasible or i must only bond hardware cards?

Is the bonding network aware?
If the ib cable is connected to the ib switch but the switch is disconnected from the LAN will it fall-back to ethernet?


Thanks in advance

 

[plokko]

I tried various configuration, bonding one ib with one ethernet as active-backup does not work:
even if the ib network is set as default only the ethernet nework will be used.
As much as i can tell ipoib does not work well with bonding and does not work at all bonded with other network types:
is there a way to redirect network traffic in some other ways?

 

[Stoiko Ivanov]

As redundancy is very important i want to use the infinibad networking as the main network and the ethernet cards as a fallback if a cable is disconnected or the ib switch is down.

I'm not sure that a fallback from 40g ipoib to 1g ethernet will do you too much good (at least not if you're using >1g throughput at some point, which is likely with ceph).

I don't have too much experience with IB, but a quick search regarding bonding on IB suggests that active-backup is the only supported mode (802.3ad is an ethernet-standard after all) - see e.g. https://community.mellanox.com/s/ar...-bond--lag--interface-over-infiniband-network (RH/CentOS specific config-examples but the basics should apply).

I hope this helps!

 

[plokko]

I'm not sure that a fallback from 40g ipoib to 1g ethernet will do you too much good (at least not if you're using >1g throughput at some point, which is likely with ceph).

I don't have too much experience with IB, but a quick search regarding bonding on IB suggests that active-backup is the only supported mode (802.3ad is an ethernet-standard after all) - see e.g. https://community.mellanox.com/s/ar...-bond--lag--interface-over-infiniband-network (RH/CentOS specific config-examples but the basics should apply).

I hope this helps!

The 1Gb network is bound to have less than ideal performance but at least it provides a backup connection to avoid single point of faiure (if the infiniband switch goes down).

I already found the article you mentioned and, as much as i can tell from my tests, it seems the only viable option is to use a redundant IB switch due to the fact that bonding ipoib seems unfeasible (in my tests the logs showed that ethernet network was used instead of ib even if ib was available and the preferred option).

Just to be sure are there any other way to create am backup network other than bonding? (ex. specifying a backup network in Ceph or via Linux bridge/OVS bond)

 

[Stoiko Ivanov]

Honestly haven't tried to bond 2 bonds together - could imagine that bonding 1 ib interface with 1 ethernet interface does not work.

Theoretically creating on active-backup ib bond should work (start with that)
1 802.3ad bond over ethernet should work! (if your ethernet-switch supports it) - active-backup should work in any case
Theoretically an active-backup bond of bonds could also work - haven't tried....

the config you posted initially seems to be a mix and match of RH/CentOS and Debian config the
BONDING_OPTS line does not work with /etc/network/interfaces! - Check debian's wiki https://wiki.debian.org/Bonding (PVE uses /etc/network/interfaces, not systemd-networkd)- the kernel docs on bonding https://www.kernel.org/doc/Documentation/networking/bonding.txt
and our reference documentation https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration

Regarding the fail-over from 10g to 1g - In my experience a network using more than 1 g (which ceph usually does if possible and if you want to have any performance) when being restricted to 1g, does not work slower, it stops working at all.

Regarding redundancy on a higher level (layer 3, or application-layer) - theoretically you could create a equal-cost-routing with frr, but that is far more complicated and needs quite some knowledge and I would not recommend it as an alternative to a redundant switch (or 2 10g ethernet NICs).