Bridge with 1gb and 10gb ethernets?

[delebru]

What should I expect if I create a bridge with a 1gb port, a 10gb port and connect both to the same switch? Will it automatically use the 10gb link for higher speeds?

I have an annoying case that when our server gets rebooted, we need to manually disconnect and reconnect the DAC cable to the switch or Mellanox SFP+ port for the 10gb link to start working.
The server needed to be turned off early on Saturday for electrical work at the premises, I asked the electrician to turn everything back on but forgot about that issue... now I won’t be able to connect until Monday when I come to the office.

I thought making the 1+10gb bridge would solve this situation but would it bring me any other problems?

 

[LnxBil]

What should I expect if I create a bridge with a 1gb port, a 10gb port and connect both to the same switch? Will it automatically use the 10gb link for higher speeds?

Just bridging will most likely kill your network, because you created a loop in a bridge.

I thought making the 1+10gb bridge would solve this situation but would it bring me any other problems?

I suggest you use a bonded device with the two devices and add it to the bridge. This is normal operation in high availability settings, where you have multiple redundant network paths. I suggest to use active-backup bonding, that works will all switches.

I have an annoying case that when our server gets rebooted, we need to manually disconnect and reconnect the DAC cable to the switch or Mellanox SFP+ port for the 10gb link to start working.

Have you tried a ifdown and ifup?

 

[spirit]

What should I expect if I create a bridge with a 1gb port, a 10gb port and connect both to the same switch? Will it automatically use the 10gb link for higher speeds?

I have an annoying case that when our server gets rebooted, we need to manually disconnect and reconnect the DAC cable to the switch or Mellanox SFP+ port for the 10gb link to start working.
The server needed to be turned off early on Saturday for electrical work at the premises, I asked the electrician to turn everything back on but forgot about that issue... now I won’t be able to connect until Monday when I come to the office.

I thought making the 1+10gb bridge would solve this situation but would it bring me any other problems?

do an active-backup bond, and make 10gb port as primary.

 

[delebru]

Many thanks for the replies! Active backup bonding is exactly what I need here, I'm glad I didn't try just adding another interface to the bridge first

Through the web UI I created the bond and added the 10gb interface (enp65s0) and a 1gb interface (eno1) but regardless of which one I type in first, eno1 stays on the first position. Will that make eno1 as the primary adapter of the bond?

This is how my interfaces.new looks like:

iface eno1 inet manual
#Onboard 1Gbe #1

iface enp65s0 inet manual
#10Gbe SFP+

auto bond0
iface bond0 inet manual
    bond-slaves eno1 enp65s0
    bond-miimon 100
    bond-mode active-backup
#Redundant aggregation

auto vmbr0
iface vmbr0 inet static
    address  192.168.10.3
    netmask  255.255.255.0
    gateway  192.168.10.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
#Main Network

Should I just replace " bond-slaves eno1 enp65s0" with " bond-slaves enp65s0 eno1"? Or is there another way to define the primary adapter?

Have you tried a ifdown and ifup?

I tried but on the bridge which brought up the network on the host but I had to restart all containers. I'll try next time on the actual interface.
The problem with this, is that I can't do it remotely when the host is stuck with no network after a reboot.

 

[spirit]

you can add
"
bond-primary enp65s0
"

option , in bond interface

 

[delebru]

bond-primary enp65s0

Thanks!

I gave that config a try last night but after rebooting I had all sorts of network issues, it seemed to be switching back and forth from one interface to the other. Ended up reverting to no bond and with enp65s0 directly on the bridge until I can find out what is going on.

Just now I realized that only the bridge and bond have the auto start enabled. Do the interfaces need to auto start as well before the bond and bridge?

 

[LnxBil]

Just now I realized that only the bridge and bond have the auto start enabled. Do the interfaces need to auto start as well before the bond and bridge?

The dependencies should be taken care of and it should just work. This is our working configuration:

auto bond0
iface bond0 inet manual
        slaves eth0 eth1
        bond_miimon 100
        bond_mode 1

auto vmbr0
iface vmbr0 inet static
        address x.y.z
        netmask 255.255.0.0
        gateway x.y.z
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0

We do not have any other configuration (e.g. eth0 and eth1) inside. The host is an updated PVE3->PVE4->PVE5

 

[delebru]

Thanks for the help! And yes, that's how my config looked like. Only the bond and bridge on auto start, but no luck.

I'm wondering if my issues may be related to the ixgbe driver. I had to add

options ixgbe allow_unsupported_sfp=1

to /etc/modprobe.d/ixgbe.conf for the interface to work but maybe it needs something else to work at boot time? I read somewhere that adding

GRUB_CMDLINE_LINUX=" ixgbe.allow_unsupported_sfp=1"

@ /etc/default/grub would do the trick. I might have time on the weekend to give this a try... Any thoughts?

 

[LnxBil]

IIRC, the modprobe stuff is also copied to the initrd, so an update of that should do the trick.

 

[Stoiko Ivanov]

or Mellanox SFP+ port for the 10gb

options ixgbe allow_unsupported_sfp=1

the ixgbe options are only for the ixgbe module (which handles Intel 10g nics (X520 and related)) - and not mellanox!
(I hope I did not miss something while reading)

I would check `dmesg` and the `journalctl` output for issues with the network-card to find out why the unplug+plug is necessary

 

[delebru]

IIRC, the modprobe stuff is also copied to the initrd, so an update of that should do the trick.

Cool, thanks, I did run update-initramfs -u.

the ixgbe options are only for the ixgbe module (which handles Intel 10g nics (X520 and related)) - and not mellanox!
(I hope I did not miss something while reading)

Nah, you didn’t miss anything. But it was the only way I found to make the interface work (it's connected with a Ubiquiti DAC to a US-48). Maybe there's a similar option for mellanox drivers?

Now that I think about it, I'm wondering if the unplug/replug would have worked without adding the ixgbe line... Will do some tests tonight.

 

[LnxBil]

Now that I think about it, I'm wondering if the unplug/replug would have worked without adding the ixgbe line... Will do some tests tonight.

Have you tried to get it to work with ethtool, e.g. manually setting the wrong speed and then going back to auto. I had a similar problem I solved with this. Reloading the network driver for the specific card could also help.

 

[delebru]

So back to square 1: the mellanox works fine on its own (no bridge, only the interface setup on vmbr0) without the ixgbe setting. Bridge comes up after boot and all vms are reachable. Not sure why I added that setting on the first place...

I tried creating the bond again (active backup with sfp+ and 1gbe, sfp+ as primary), rebooted and everything seemed to work fine but iptraf showed the traffic being split between eno1 and enp65s0. Then disconnected the dac cable from the mellanox and it seemed to transfer nicely to the other ethernet, took a few seconds but worked. Then reconnected the dac and everything went to... horrible

I think I’m going to leave it with only the mellanox directly on the bridge now that is working after reboot. Thanks everyone for your help!!!

I’d love to understand/fix this bond but it’s not enjoyable trying to figure this out without a testing platform.

 

[LnxBil]

active/passive bond with preferred nic should do the trick.

I know, you reconfigured it, but if you try again, please answer:
- How did you configure your bond?
- Post contents of /proc/net/bonding/bond0

 

[delebru]

active/passive bond with preferred nic should do the trick.

I know, you reconfigured it, but if you try again, please answer:
- How did you configure your bond?
- Post contents of /proc/net/bonding/bond0

I setup the bond again just to answer your questions... I did the config on the web UI, then only added bond-primary enp65s0 to /etc/network/interfaces.new and reboot. Everying as I thought I had done before, but now iptraf is only showing traffic on the mellanox... Not sure what I could have done wrong the first time... I'll leave it set up for now, might do some disconnect/re-connect tests again when I have some time.

~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: enp65s0 (primary_reselect always)
Currently Active Slave: enp65s0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eno1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: c8:1f:66:f2:7d:2c
Slave queue ID: 0

Slave Interface: enp65s0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:02:c9:54:68:20
Slave queue ID: 0

~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual
iface enp65s0 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 enp65s0
        bond-primary enp65s0
        bond-miimon 100
        bond-mode active-backup

auto vmbr0
iface vmbr0 inet static
        address  192.168.10.3
        netmask  255.255.255.0
        gateway  192.168.10.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

 

[LnxBil]

Looks ok. If you unplug/replug the network devices, you should see an increase in link failures
(/proc/net/bonding/bond0) but everything should work as expected. I'm running this configuration on hundreds of servers without any issue for decades.

if you want to change the device on the fly:

ifenslave -c bond0
eno1