No balancing out of Promox node with LACP 802.3ad

[Dulcow]

Hi there,

I'm trying to test the various bonds that I have configured. I will try to be as exhaustive as I can be for the setup. I have 4x nodes in total:

1. NAS bonded to MikroTik CRS317 (2x10G SFP+)
2. MikroTik CRS317 bonded to MikroTik CRS310 (2x10G SFP+)
3. Each of the 3x NUCs bonded to MikroTik CRS310 (2x2.5G)

On the switch side, all the bonds are configured the same way:



On Proxmox side, all the nodes are configured the same way as well:

root@pve-nuc12-3:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp113s0
iface enp113s0 inet manual

auto enp114s0
iface enp114s0 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp113s0 enp114s0
        bond-miimon 100
        bond-mode 802.3ad
        bond-downdelay 200
        bond-updelay 200
        bond-xmit-hash-policy layer3+4
        bond-lacp_rate fast

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.213/24
        gateway 192.168.10.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

Balancing traffic in (NAS > NUC)
Using 2x iperf clients on the NAS to 2x iperf servers with different ports on the NUC => This works well, I can see a total of 4.8G in the switch UI.

Balancing traffic out (NUC > NAS)
Using 2x iperf clients on the NUC to 2x iperf servers with different ports on the NAS => This does not work as traffic goes out via the same network interface on the NUC. The overall bandwidth is shared.

Any idea what is going here?

Thanks,

D.

 

[gurubert]

bond-xmit-hash-policy laye3+4

You are missing an "r" here.

 

[Dulcow]

FFS... Thank you very much!

Several people went through this file though

 

[Dulcow]

Well, I spoke too fast... Even with the typo removed, it still does not work from the NUC to the NAS :-(

 

[Dulcow]

It has worked for a few days and now, it is broken again. It works fine through the switch (NAS > NUC), still half bandwidth the other way around (NUC > NAS).

A bit more details about the setup :

root@pve-nuc12-3:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.5.13-5-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 48:21:0b:60:3e:a7
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 11
        Partner Key: 11
        Partner Mac Address: d4:01:c3:0e:5b:c0

Slave Interface: enp114s0
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 48:21:0b:60:3e:a7
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: 48:21:0b:60:3e:a7
    port key: 11
    port priority: 255
    port number: 1
    port state: 63
details partner lacp pdu:
    system priority: 65535
    system mac address: d4:01:c3:0e:5b:c0
    oper key: 11
    port priority: 255
    port number: 1
    port state: 63

Slave Interface: enp115s0
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 48:21:0b:58:cc:c2
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: 48:21:0b:60:3e:a7
    port key: 11
    port priority: 255
    port number: 2
    port state: 63
details partner lacp pdu:
    system priority: 65535
    system mac address: d4:01:c3:0e:5b:c0
    oper key: 11
    port priority: 255
    port number: 2
    port state: 63

root@pve-nuc12-3:~# lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    enp114s0, via: LLDP, RID: 1, Time: 0 day, 00:00:05
  Chassis:     
    ChassisID:    mac d4:01:c3:0e:5b:bc
    SysName:      MikroTik
    SysDescr:     MikroTik RouterOS 7.15 (stable) 2024-05-29 12:44:08 CRS310-8G+2S+
    MgmtIP:       fe80::d601:c3ff:fe0e:5bc2
    MgmtIface:    6
    Capability:   Bridge, on
    Capability:   Router, on
  Port:       
    PortID:       ifname ether5
    PortDescr:    bridge1/bondnuc3/ether5
    TTL:          120
-------------------------------------------------------------------------------
Interface:    enp115s0, via: LLDP, RID: 1, Time: 0 day, 00:00:03
  Chassis:     
    ChassisID:    mac d4:01:c3:0e:5b:bc
    SysName:      MikroTik
    SysDescr:     MikroTik RouterOS 7.15 (stable) 2024-05-29 12:44:08 CRS310-8G+2S+
    MgmtIP:       fe80::d601:c3ff:fe0e:5bc2
    MgmtIface:    6
    Capability:   Bridge, on
    Capability:   Router, on
  Port:       
    PortID:       ifname ether6
    PortDescr:    bridge1/bondnuc3/ether6
    TTL:          120
-------------------------------------------------------------------------------

These NUCs are using the infamous i225v and i226v Intel NICs. Could it be related? Something funky with the drivers?

If someone has an idea on how to troubleshoot / move forward, please throw everything at me ;-)

Thanks,

G.

 

[gurubert]

Something funky with the drivers?

Could be. This sounds really strange.

 

[Dulcow]

Could be. This sounds really strange.

Would you think about other things that I forgot to enable on Proxmox side? I think it is client related as the the traffic coming to the Mikrotik switch has the right expected behaviours.