Bond Interface not working

[RNab]

Hi all,

I'm struggling to create a LACP bond interface in Proxmox. I'm trying to aggregate two 1Gb NIC to increase bandwidth.
With the config described below, if I connect only of the ethernet, then it works fine, but when they are both connected, it fails to ping or reach Proxmox.


I'm detailing my Proxmox /etc/network/interfaces below :

auto lo
iface lo inet loopback

iface enp89s0 inet manual

#auto enxfc34971ec164
iface enxfc34971ec164 inet manual

auto bond0
iface bond0 inet manual
        bond-primary enp89s0
        bond-slaves enp89s0 enxfc34971ec164
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.2/24
        gateway 192.168.0.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

 

[Dunuin]

Try it without "bond-primary enp89s0" in your config.

 

[RNab]

Try it without "bond-primary enp89s0" in your config.

Yes actually I just added bond-primary enp89s0 5 minutes ago as I saw someone suggesting it in the forum.
Without it, same problem

Just to add one info : after removing/adding it, and after ifreload -a, I have this warning :

warning: bond0: attribute bond-min-links is set to '0'

Not sure the meaning of it though

 

[Dunuin]

And your switch supports LACP and is also set to "layer2+3"?

 

[RNab]

I’m not too sure actually. I realised my switch manual mentions only LAG and static link aggregation.

I don’t know enough about LACP and LAG.

https://cdn.cnetcontent.com/77/7c/777c1e16-24f7-4890-8ee1-69f16bdec540.pdf

 

[Dunuin]

You need to bond those ports on the switch aswell and the hashing algorithm has to match. The datasheet didn't explicitly mentions IEEE 802.3ad support which is LACP.

 

[RNab]

Ok thanks for your help.

From what you see on the data sheet, anything I can do to improve my network speed ? Or is the only solution a new switch ?

 

[Donovan Hoare]

I’m not too sure actually. I realised my switch manual mentions only LAG and static link aggregation.

I don’t know enough about LACP and LAG.

https://cdn.cnetcontent.com/77/7c/777c1e16-24f7-4890-8ee1-69f16bdec540.pdf

I don't know this switch but from what you posted it looks like it can.
Have you logged into the switch to configure the LACP Bind on the ports?

I normally use Cisco so (EtherChannel)
or Mikrotik - Bonds.
You should be able to figure it out there.

 

[RNab]

Yes indeed, the switch is configured for the two ports to be on LAG (id 2)

 

[Donovan Hoare]

Your config and mine look slightly different.
The below is a old one of mine as i now use openvswitch the next config as a sample my management interface is vlan 11

auto bond0
iface bond0 inet manual
                slaves eno1,eno2
                bond_miimon 100
                bond_mode 802.3ad
                bond_lacp_rate 1
                mtu 9000

auto eno1
iface eno1 inet manual
                bond-master bond0
                mtu 9000
 
auto eno2
iface eno2 inet manual
                bond-master bond0
                mtu 9000
 
auto eno3
iface eno3 inet static
                address 192.168.28.15
                netmask 255.255.252.0
                mtu 9000

auto eno4
iface eno4 inet manual
                mtu 9000
 
auto vmbr0
iface vmbr0 inet manual
                bridge_ports bond0
                bridge_stp off
                bridge_fd 0
                bridge_vlan_aware yes
                mtu 9000
 
auto bond0.11
iface bond0.11 inet manual
                vlan-raw-device bond0
                mtu 9000
 
auto vmbr0v11
iface vmbr0v11 inet static
                bridge_ports bond0.11
                bridge_stp off
                bridge_fd 0
                bridge_vlan_aware yes
                mtu 1500
                address 192.168.241.15
                netmask 255.255.255.0
                gateway 192.168.241.1

This is an openvswitch example
in this xample my management vlan is 20
if you want to use openvswitch
you must remember to install it 1st

# Loopback interface
auto lo
iface lo inet loopback

# Bond eno3 and eno4 together
auto eno3
iface eno3 inet manual
    ovs_mtu 9000

auto eno3
iface eno3 inet manual
    ovs_mtu 9000

auto bond0
iface bond0 inet manual
  ovs_bridge vmbr0
  ovs_type OVSBond
  ovs_bonds eno3 eno4
  #pre-up ( ifconfig enp7s0f0 mtu 9000 && ifconfig enp7s0f1 mtu 9000 )
  ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast tag=1 vlan_mode=native-untagged
  #ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast tag=1 vlan_mode=native-untagged
  ovs_mtu 9000
  #mtu 9000

  # Bridge for our bond and vlan virtual interfaces (our VMs will
# also attach to this bridge)
auto vmbr0
iface vmbr0 inet manual
  ovs_type OVSBridge
  # NOTE: we MUST mention bond0, vlan50, and vlan55 even though each
  #       of them lists ovs_bridge vmbr0!  Not sure why it needs this
  #       kind of cross-referencing but it won't work without it!
  ovs_ports bond0 vlan20
  ovs_mtu 9000
  #mtu 9000

auto vlan20
iface vlan20 inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr0
    ovs_options tag=20
    #ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
    address 192.168.14.15
    netmask 255.255.254.0
    gateway 192.168.14.1
    ovs_mtu 1500
    #mtu 1500

 

[RNab]

I tried your first config (slightly adjusted to my config), and I'm getting this error :



warning: bond: attribute bond-min-links is set to '0'
error: netlink: bond: cannot set dev bond address 10:69:7a:a5:04:cd: operation failed with 'Device or resource busy'

This is what the config looks like exactly

auto bond0
iface bond0 inet manual
        bond-slaves enp89s0 enxfc34971ec164
        bond miimon 100
        bond_mode 802.3ad
        bond_lacp_rate 1
        mtu 1500

auto enp89s0
iface enp89s0 inet manual
        bond-master bond0
        mtu 1500

auto enxfc34971ec164
iface enxfc34971ec164 inet manual
        bond-master bond0
        mtu 1500

auto vmbr0
iface vmbr0 inet manual
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        bridge_vlan_aware yes
        mtu 1500

auto bond0.3
iface bond0.3 inet manual
        vlan-raw-device bond0
        mtu 1500

auto vmbr0v3
iface vmbr0v3 inet static
        bridge_ports bond0.3
        bridge_stp off
        bridge_fd 0
        bridge_vlan_aware yes
        mtu 1500
        address 10.10.30.2/24

 

[Donovan Hoare]

That is an old conf. i Had to update my ovs configs due to the change to ifupdown2
maybe there is a slight adjustment that can help.

Perhaps soneelse knows

 

[bzb-rs]

Just to throw in here: i am also having some hardtime getting lacp 802.3ad <any-mode> to get it working. Currently setup balance-xor mode until i get this sorted.

We have 3 other servers with exact same config and they all work perfectly fine.

auto lo
iface lo inet loopback

auto eno1np0
iface eno1np0 inet manual

auto eno2np1
iface eno2np1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1np0 eno2np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address *
        gateway *
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

 

[emunt6]

Hi!
I was having the same problem, here is the fix:

/etc/modules-load.d/modules.conf
-------------------------------------------------------
options bonding max_bonds=0
-------------------------------------------------------

if you make any changes in the "/etc/network/interfaces" file, and you need to run the following too:

$> update-initramfs -c -d -u

( if yo not run this, initramfs does not get updated - the old "interfaces" file will kept inside )