Struggling with network bond

[hefferbub]

I'm trying to set up a simple network bond, and I can't seem to get it to work. My server has 2 NICs which I am trying to connect to an HP1920/48G switch for increased network capacity. I would like the server to present a single IP address for management and have the VMs use the same bridge for their access, all on the same subnet.

On the switch side, I created a 2-port static link aggregation using 802.3ad, and I can see that the switch considers both ports "selected"--that is active and part of the link aggregation.

On the server side (proxmox 4.4-5) I have the following in /etc/network/interfaces:

auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth1 inet manual

auto bond0
iface bond0 inet manual
    slaves eth0 eth1
    bond_miimon 100
    bond_mode 802.3ad
    bond_xmit_hash_policy layer2+3

auto vmbr0
iface vmbr0 inet static
    address 192.168.240.195
    netmask 255.255.252.0
    gateway 192.168.243.24
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0

When I reboot the server, the links appear to come up, but I cannot access the server by ssh nor web interface (although weirdly I can get through under certain rare circumstances). I don't know if the VMs can communicate through the bridge or not.

Can anyone see any obvious problems with this setup and/or my config? Can you suggest a more robust way to achieve the goal?

Thanks,

Jeff

 

[manu]

Hi Jeff

Please check in this order:
Are the underlying network links seeing network traffic ?
ip link show dev eth0
ip link show dev eth1

here you should look for the string LOWER_UP indicating the NIC is seeing traffic at the Ethernet Level

Is the bond active ?
ip link show dev bond0
here you should look for the strings MASTER and LOWERUP

is the bond properly added to the bridge ?
brctl show vmbr0
here you should look for your bond0 interface in the list of devices ( I suppose)

Your config looks OK to me.

 

[hefferbub]

Thanks for those excellent diagnostic suggestions. I attached the results, which look OK to me:



The weird thing is, I can ping this IP from the switch it is connected to. The weirder thing is, I can connect to the Proxmox web interface at this IP from my iPad, but not from a Mac or Windows or iPhone connected to the same wireless router.

It must be something in my network...

 

[boopzz]

Id probably try leaving your config and unplugging one of the cables. If everything works fine then Id suggest specifically looking at your LAG config and make sure the Proxmox and Switch have a similar config for the ports that are connect.

 

[hefferbub]

Ah, I finally figured it out! On the hp 1920 switch, when I created the "Link Aggregation", I created it as "static". It should have been "dynamic" which enables the LACP protocol. Seems fine now.

Thanks for all suggestions.

 

[fishburn24]

I figure I would reply with a similar issue instead of starting a new thread. I have pve running on a Dell Optiplex 5050 with an 2 - NICs, three interfaces.

I setup bonding on the NIC with two interfaces; however, I can only get connectivity with the onboard NIC, not with either interfaces on the dual NIC! Below is my ip link show dev and /etc/network/interface configuration:

ip link show dev for each interface:

ip link show dev enp0s31f6
4: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether d8:9e:f3:21:5a:cd brd ff:ff:ff:ff:ff:ff
root@cmanpm:~# ip link show dev enp1s0f0
2: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:10:18:69:98:d4 brd ff:ff:ff:ff:ff:ff
root@cmanpm:~# ip link show dev enp1s0f1
3: enp1s0f1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:10:18:69:98:d4 brd ff:ff:ff:ff:ff:ff
root@cmanpm:~# ip link show dev vmbr0
6: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:10:18:69:98:d4 brd ff:ff:ff:ff:ff:ff

/etc/network/interface config:

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

iface enp1s0f0 inet manual

iface enp1s0f1 inet manual

auto bond0
iface bond0 inet manual
      slaves enp1s0f0 enp1s0f1
      bond_miimon 100
      bond_mode 802.3ad
      bond_xmit_hash_policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address  192.168.2.29
        netmask  255.255.255.192
        gateway  192.168.2.1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0

Thank you in advance for any assistance!

 

[spirit]

I figure I would reply with a similar issue instead of starting a new thread. I have pve running on a Dell Optiplex 5050 with an 2 - NICs, three interfaces.

I setup bonding on the NIC with two interfaces; however, I can only get connectivity with the onboard NIC, not with either interfaces on the dual NIC! Below is my ip link show dev and /etc/network/interface configuration:

ip link show dev for each interface:

ip link show dev enp0s31f6
4: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether d8:9e:f3:21:5a:cd brd ff:ff:ff:ff:ff:ff
root@cmanpm:~# ip link show dev enp1s0f0
2: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:10:18:69:98:d4 brd ff:ff:ff:ff:ff:ff
root@cmanpm:~# ip link show dev enp1s0f1
3: enp1s0f1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:10:18:69:98:d4 brd ff:ff:ff:ff:ff:ff
root@cmanpm:~# ip link show dev vmbr0
6: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:10:18:69:98:d4 brd ff:ff:ff:ff:ff:ff

/etc/network/interface config:

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

iface enp1s0f0 inet manual

iface enp1s0f1 inet manual

auto bond0
iface bond0 inet manual
      slaves enp1s0f0 enp1s0f1
      bond_miimon 100
      bond_mode 802.3ad
      bond_xmit_hash_policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address  192.168.2.29
        netmask  255.255.255.192
        gateway  192.168.2.1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0

Thank you in advance for any assistance!

seem than enp1s0f0/1 are down. Are you sure to have configured lacp on your physical switch ?

can you send result of

"cat /proc/net/bonding/bond0 " ?

 

[fishburn24]

Note, I changed up the slave interfaces to test. I did confirm I am using a switch that does not support lacp. I do have an open port on my utm that I can configure to use lacp - I presume this is what is MII capable means? Otherwise, this is really only a test machine and currently only running a secondary dns so I really don't have to have link aggregation. I presume I can set it back to the standard single bridged configuration; however, only use the onboard NIC? Thank you.

result of cat /proc/net/bonding/bond0 :

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 00:10:18:69:98:d4
bond bond0 has no active aggregator

Slave Interface: enp1s0f0
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 1
Permanent HW addr: xx:xx:xx:xx:xx:xx
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 2
details actor lacp pdu:
    system priority: 65535
    system mac address: xx:xx:xx:xx:xx:xx
    port key: 0
    port priority: 255
    port number: 1
    port state: 77
details partner lacp pdu:
    system priority: 65535
    system mac address: 00:00:00:00:00:00
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

Slave Interface: enp0s31f6
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: xx:xx:xx:xx:xx:xx
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: xx:xx:xx:xx:xx:xx
    port key: 0
    port priority: 255
    port number: 2
    port state: 69
details partner lacp pdu:
    system priority: 65535
    system mac address: 00:00:00:00:00:00
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

 

[zizzithefox]

Actually I have a similar issue connecting two proxmox servers.
I was trying to increase performance by bonding the two 1 Gbit interfaces of my virtualized truenas server (let's call it truenas on proxmox1). These two ports are connected to another proxmox server (proxmox2) that has six 2.5Gbit ports, 2 of which I bonded together in the same way.
In both cases the bonds are added to a proxmox switch in order for the bond to be shared by the virtual machines on the respective servers.

As long as the proxmox servers are communicating with each other, it works in balance-rr mode, because if I call iperf from the proxmox1 to proxmox2, it sustains 1.5Gbit/s with both cables, and 950Mbit/s if I unplug one of them, so fault tolerance is also working.

Problem is that, as I run iperf from the truenas on proxmox1 to proxmox2, both cables plugged in, I get a ridiculuos 300Mbit/s. If I just unplug one of them, doesn't matter which one, I still get 950 Mbit/s.

Weird. I will try to set up routing on proxmox2, when I feel like it, just to see if something changes. But, as expected, speed would be too much slower anyway, when compared to a proper 2.5Gbit link. So, I guess a pcie card is in order.