redundant separate 10GBit network

sannsio

Member
Dec 8, 2015
27
2
23
Hi,

I would like to implement more redundancy to my pve/ceph cluster. The storage runs in a separate 10 Gbit network. I was thinking to add a second 10 Gbit switch and connect the serves NIC's to each of them using bonding with active-backup mode.

Is it advisable to do that? Is active-passive bonding reliable and fast enough? Or will packets get lost and cause storage inconsistency when a NIC or a switch fails?

Thanks for any advise!
Sandra
 
Dec 13, 2018
48
4
13
50
We use active / backup bonding for everything except corosync networks.

It works well and hasn't caused any issues. We've had reason to bring a switch offline (firmware updates, etc.) and used these reasons as a test. We still schedule these offline periods for after hours but have never had an issue with the primary / backup links coming online / going offline ... they just work.
 

sannsio

Member
Dec 8, 2015
27
2
23
We use active / backup bonding for everything except corosync networks.

It works well and hasn't caused any issues. We've had reason to bring a switch offline (firmware updates, etc.) and used these reasons as a test. We still schedule these offline periods for after hours but have never had an issue with the primary / backup links coming online / going offline ... they just work.

Hey,
thanks for your feedback! Sounds promising :)

Have you also been using Linux Bridges for redundancy? I was reading about it and some people are saying that using bridges connected to multiple switches (using spanning tree) is the better solution than active-backup bonding.

Cheers,
Sandra
 
Dec 13, 2018
48
4
13
50
Bridges? No, I haven't used this before. That sounds interesting.

Our current setup only uses STP for the physical switch trunks.

Hmm ... even RSTP may take more time that you would like to detect a failure and converge. Also, if you add VLANs to the mix do your switches support PVST?

If you can afford switches that support MLAG this would probably be a better option than what you are proposing or what I am using.
 

sannsio

Member
Dec 8, 2015
27
2
23
I will try the active-passive bonding method. I think in small separated segments like a storage network, it is a good solution.
Maybe bridging is the better approach when connecting servers to a wider network segment were STP is in place? Don't know...

Maybe someone else here can add his opinion/experience. Would be interesting to see what solutions are in place.

Thanks again!
Sandra
 

alexskysilk

Well-Known Member
Oct 16, 2015
728
86
48
Chatsworth, CA
www.skysilk.com
If I only have one pair of links for cluster traffic, I create two active/backup bonds with the same links (different vlans) with alternate masters. one is used for ceph, one is used for corosync. While it is better to have corosync on two non bonded interfaces, this allows me to maintain full functionality, redundancy, with relatively good performance as both traffic types remain on their own physical interfaces during normal operation.
 

jinjer

Active Member
Oct 4, 2010
203
6
38
If I only have one pair of links for cluster traffic, I create two active/backup bonds with the same links (different vlans) with alternate masters. one is used for ceph, one is used for corosync.
Would you share your /etc/network/interfaces configuration ?
If I understand exactly, you managed to bond the two links like this:
Active-Backup - bond0
Backup-Active - bond1

Then, on a switch failure (say the second switch), they become :

Active-Backup - bond0
Active-Backup - bond1

At the same time you have a bond0.x and bond1.y vlans on top of the bond ?

Did I understand correctly? Is it possible to enslave an interface to 2 bonds?

Thank you.
 

alexskysilk

Well-Known Member
Oct 16, 2015
728
86
48
Chatsworth, CA
www.skysilk.com
Would you share your /etc/network/interfaces configuration ?

Code:
auto bond1
iface bond1 inet static
        address  10.0.0.10
        netmask  255.255.255.0
        slaves eth0.1 eth1.1
        bond-primary eth0
        bond_miimon 100
        bond_mode active-backup
        mtu 9000

auto bond2
iface bond2 inet static
        address  10.1.0.10
        netmask  255.255.255.0
        slaves eth1.2 eth0.2
        bond-primary eth1
        bond_miimon 100
        bond_mode active-backup
        mtu 9000
active/backup refers to the link aggregation method. it is implied that at any time traffic flows across a single interface, not both. In the above example, bond1 travels on vlan 1 with eth0 as its preferred master- if eth0 is available traffic will flow over it and not the other. bond2 travels on vlan 2 with eth1 as its preferred master. In case of link fault, both bonds will have traffic travel on the surviving interface regardless of preferred master.

It is also possible (and maybe preferrable) to have ceph traffic over a bonded interface, but corosync traffic seperated to two rings instead.
 

jinjer

Active Member
Oct 4, 2010
203
6
38
Thank you. I was under the false assumption that you cannot enslave a vlan interface (as was the case some years ago).
I was not able to replicate your setup and confirm it works nicely accross multiple switches for speed and as a failover if a whole switch dies.
Further, connecting the switches via lag groups it's possible to sustain a single-link failure on a bond with only performance degradation.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!