redundant separate 10GBit network

[sannsio]

Hi,

I would like to implement more redundancy to my pve/ceph cluster. The storage runs in a separate 10 Gbit network. I was thinking to add a second 10 Gbit switch and connect the serves NIC's to each of them using bonding with active-backup mode.

Is it advisable to do that? Is active-passive bonding reliable and fast enough? Or will packets get lost and cause storage inconsistency when a NIC or a switch fails?

Thanks for any advise!
Sandra

 

[Binary Bandit]

We use active / backup bonding for everything except corosync networks.

It works well and hasn't caused any issues. We've had reason to bring a switch offline (firmware updates, etc.) and used these reasons as a test. We still schedule these offline periods for after hours but have never had an issue with the primary / backup links coming online / going offline ... they just work.

 

[sannsio]

We use active / backup bonding for everything except corosync networks.

It works well and hasn't caused any issues. We've had reason to bring a switch offline (firmware updates, etc.) and used these reasons as a test. We still schedule these offline periods for after hours but have never had an issue with the primary / backup links coming online / going offline ... they just work.

Hey,
thanks for your feedback! Sounds promising

Have you also been using Linux Bridges for redundancy? I was reading about it and some people are saying that using bridges connected to multiple switches (using spanning tree) is the better solution than active-backup bonding.

Cheers,
Sandra

 

[Binary Bandit]

Bridges? No, I haven't used this before. That sounds interesting.

Our current setup only uses STP for the physical switch trunks.

Hmm ... even RSTP may take more time that you would like to detect a failure and converge. Also, if you add VLANs to the mix do your switches support PVST?

If you can afford switches that support MLAG this would probably be a better option than what you are proposing or what I am using.

 

[sannsio]

I will try the active-passive bonding method. I think in small separated segments like a storage network, it is a good solution.
Maybe bridging is the better approach when connecting servers to a wider network segment were STP is in place? Don't know...

Maybe someone else here can add his opinion/experience. Would be interesting to see what solutions are in place.

Thanks again!
Sandra

 

[alexskysilk]

If I only have one pair of links for cluster traffic, I create two active/backup bonds with the same links (different vlans) with alternate masters. one is used for ceph, one is used for corosync. While it is better to have corosync on two non bonded interfaces, this allows me to maintain full functionality, redundancy, with relatively good performance as both traffic types remain on their own physical interfaces during normal operation.

 

[jinjer]

If I only have one pair of links for cluster traffic, I create two active/backup bonds with the same links (different vlans) with alternate masters. one is used for ceph, one is used for corosync.

Would you share your /etc/network/interfaces configuration ?
If I understand exactly, you managed to bond the two links like this:
Active-Backup - bond0
Backup-Active - bond1

Then, on a switch failure (say the second switch), they become :

Active-Backup - bond0
Active-Backup - bond1

At the same time you have a bond0.x and bond1.y vlans on top of the bond ?

Did I understand correctly? Is it possible to enslave an interface to 2 bonds?

Thank you.

 

[alexskysilk]

Would you share your /etc/network/interfaces configuration ?

auto bond1
iface bond1 inet static
        address  10.0.0.10
        netmask  255.255.255.0
        slaves eth0.1 eth1.1
        bond-primary eth0
        bond_miimon 100
        bond_mode active-backup
        mtu 9000

auto bond2
iface bond2 inet static
        address  10.1.0.10
        netmask  255.255.255.0
        slaves eth1.2 eth0.2
        bond-primary eth1
        bond_miimon 100
        bond_mode active-backup
        mtu 9000

active/backup refers to the link aggregation method. it is implied that at any time traffic flows across a single interface, not both. In the above example, bond1 travels on vlan 1 with eth0 as its preferred master- if eth0 is available traffic will flow over it and not the other. bond2 travels on vlan 2 with eth1 as its preferred master. In case of link fault, both bonds will have traffic travel on the surviving interface regardless of preferred master.

It is also possible (and maybe preferrable) to have ceph traffic over a bonded interface, but corosync traffic seperated to two rings instead.

 

[jinjer]

Thank you. I was under the false assumption that you cannot enslave a vlan interface (as was the case some years ago).
I was not able to replicate your setup and confirm it works nicely accross multiple switches for speed and as a failover if a whole switch dies.
Further, connecting the switches via lag groups it's possible to sustain a single-link failure on a bond with only performance degradation.