20G bond stuck at 10G (Intel x540-t2 + Cisco c3850)

C

crb3rob

New Member
Nov 15, 2023
9
2
3
Dear all I have a problem and I have looked everywhere.

I am setting up a cluster of 3 nodes: SYS 4028 (S3ai) + dell 630 (S3lab) + dell 740xd (S3main) using a bond (2x10G for every node) to connect them.
My switch is a CISCO catalyst 3850.

PROBLEM: iperf3 gets max 10G speed and on every node I get the following warning:

Code: 
bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond

Every node has a bond0 with 2 x 10G ports in the bond. Here the one of the configuration (they are all the same):
 
Last edited:
I have found that the problem was on the bond mode. It must be layer 3 for different ips and layer 4 for different ports
 
Actually I still have a problem with one node where I have 2 interfaces bonded.
I am getting half of the speed around 10.3 G instead of 20G.

On the bond I have two VLANs and a vmbr for proxmox management.

Does the proxmox management vmbr slow down the bond?
 
You need LACP/IEEE 802.3ad bonding on each host AND on the 2 corresponding ports onto the switch per host.
iperf2 has multithreaded tests but at the end transfer data fromand to 2 hosts from 1 to verify you are able to reach >1,5GB/s (so more than 10Gbit/s).
 
Yes. I am launching iperf3 from 2 different servers (s3main and s3lab) to the "target" server (s3ai). All the servers have a bond of 2 x 10G. The other servers can get up to 20G but the target server is getting max 10.2G/10.3G max. I have tried multiple ways:
_ with the parallel iperf3 option (-P 10)
_ launching iperf3 against different VMs of the target or against different ports

it seems that speed is capped to 10G.

All the settings are the same except that on the bond of the target there is also the vmbr used for managing proxmox. So... can this be slowing down the bond?
 
By the way here some of the config:

ethtool -i enp10s0f0



Code: 
root@S3ai:~# ethtool -i enp10s0f0

driver: ixgbe

version: 5.21.5

firmware-version: 0x80000389

expansion-rom-version:

bus-info: 0000:0a:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

root@S3ai:~# ethtool -i enp10s0f1

driver: ixgbe

version: 5.21.5

firmware-version: 0x80000389

expansion-rom-version:

bus-info: 0000:0a:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

root@S3ai:~#



I have four interfaces:

Code: 
root@S3ai:~# lshw -class network -businfo

Bus info          Device      Class          Description

========================================================

pci@0000:0a:00.0  enp10s0f0   network        Ethernet Controller 10-Gigabit X540-AT2

pci@0000:0a:00.1  enp10s0f1   network        Ethernet Controller 10-Gigabit X540-AT2

pci@0000:81:00.0  enp129s0f0  network        Ethernet Controller 10-Gigabit X540-AT2

pci@0000:81:00.1  enp129s0f1  network        Ethernet Controller 10-Gigabit X540-AT2



enp129s0f0 and enp129s0f1 are working in a bond with the ethernet card provided natively with the server. Bond seems to be slower 10G instead of 20G but working.



enp10s0f0 and enp10s0f1 do not work and it says NO-CARRIER



ip a



Code: 
root@S3ai:~# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host noprefixroute

       valid_lft forever preferred_lft forever

2: enp10s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000

    link/ether a0:36:9f:1a:45:84 brd ff:ff:ff:ff:ff:ff

3: enp10s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000

    link/ether a0:36:9f:1a:45:86 brd ff:ff:ff:ff:ff:ff

4: enp129s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000

    link/ether 0c:c4:7a:eb:35:a2 brd ff:ff:ff:ff:ff:ff

5: enp129s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000

    link/ether 0c:c4:7a:eb:35:a2 brd ff:ff:ff:ff:ff:ff permaddr 0c:c4:7a:eb:35:a3

6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000

    link/ether 0c:c4:7a:eb:35:a2 brd ff:ff:ff:ff:ff:ff

7: bond0.1001@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 0c:c4:7a:eb:35:a2 brd ff:ff:ff:ff:ff:ff

    inet 10.30.3.3/24 scope global bond0.1001

       valid_lft forever preferred_lft forever

    inet6 fe80::ec4:7aff:feeb:35a2/64 scope link

       valid_lft forever preferred_lft forever

8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 0c:c4:7a:eb:35:a2 brd ff:ff:ff:ff:ff:ff

    inet 10.30.1.3/24 scope global vmbr0

       valid_lft forever preferred_lft forever

    inet6 fe80::ec4:7aff:feeb:35a2/64 scope link

       valid_lft forever preferred_lft forever

9: bond0.1000@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000

    link/ether 0c:c4:7a:eb:35:a2 brd ff:ff:ff:ff:ff:ff

10: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 0c:c4:7a:eb:35:a2 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::ec4:7aff:feeb:35a2/64 scope link

       valid_lft forever preferred_lft forever



/etc/network/interfaces



Code: 
root@S3ai:~# cat /etc/network/interfaces

# network interface settings; autogenerated

# Please do NOT modify this file directly, unless you know what

# you're doing.

#

# If you want to manage parts of the network configuration manually,

# please utilize the 'source' or 'source-directory' directives to do

# so.

# PVE will preserve these directives, but will NOT read its network

# configuration from sourced files, so do not attempt to move any of

# the PVE managed interfaces into external files!



auto lo

iface lo inet loopback



auto enp129s0f0

iface enp129s0f0 inet manual

#10G



auto enp129s0f1

iface enp129s0f1 inet manual

#10G



auto enp10s0f0

iface enp10s0f0 inet manual

#10G network card not connected



auto enp10s0f1

iface enp10s0f1 inet manual

#10G network card not connected



auto bond0

iface bond0 inet manual

        bond-slaves enp129s0f0 enp129s0f1

        bond-miimon 100

        bond-mode 802.3ad

        bond-xmit-hash-policy layer3+4

        bond-downdelay 200

        bond-updelay 200

#10G aggregation



auto bond0.1000

iface bond0.1000 inet manual

#cluster public network - 10.30.2.3/24



auto bond0.1001

iface bond0.1001 inet static

        address 10.30.3.3/24

#cluster private network



auto vmbr0

iface vmbr0 inet static

        address 10.30.1.3/24

        gateway 10.30.1.1

        bridge-ports bond0

        bridge-stp off

        bridge-fd 0

#PX mgmt



auto vmbr1

iface vmbr1 inet manual

        bridge-ports bond0.1000

        bridge-stp off

        bridge-fd 0

        bridge-vlan-aware yes

        bridge-vids 2-4094



source /etc/network/interfaces.d/*



log





Code: 
[    2.248907] ixgbe: loading out-of-tree module taints kernel.

[    2.249219] ixgbe: module verification failed: signature and/or required key missing - tainting kernel

[    2.282472] ixgbe 0000:0a:00.0 0000:0a:00.0 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled

[    2.449199] ixgbe 0000:0a:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0

[    2.497404] ixgbe 0000:0a:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)

[    2.530923] ixgbe 0000:0a:00.0 eth0: MAC: 3, PHY: 3, PBA No: G45270-003

[    2.531217] ixgbe 0000:0a:00.0: a0:36:9f:1a:45:84

[    2.531453] ixgbe 0000:0a:00.0 eth0: Enabled Features: RxQ: 63 TxQ: 63 FdirHash

[    2.537877] ixgbe 0000:0a:00.0 eth0: Intel(R) 10 Gigabit Network Connection

[    2.543470] ixgbe 0000:0a:00.1 0000:0a:00.1 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled

[    2.715932] ixgbe 0000:0a:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0

[    2.765913] ixgbe 0000:0a:00.1: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)

[    2.798985] ixgbe 0000:0a:00.1 eth1: MAC: 3, PHY: 3, PBA No: G45270-003

[    2.799231] ixgbe 0000:0a:00.1: a0:36:9f:1a:45:86

[    2.799472] ixgbe 0000:0a:00.1 eth1: Enabled Features: RxQ: 63 TxQ: 63 FdirHash

[    2.805854] ixgbe 0000:0a:00.1 eth1: Intel(R) 10 Gigabit Network Connection

[    2.810660] ixgbe 0000:81:00.0 0000:81:00.0 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled

[    2.975917] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0

[    3.025016] ixgbe 0000:81:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:80:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)

[    3.033752] ixgbe 0000:81:00.0 eth2: MAC: 3, PHY: 3, PBA No: 030C00-000

[    3.034134] ixgbe 0000:81:00.0: 0c:c4:7a:eb:35:a2

[    3.034405] ixgbe 0000:81:00.0 eth2: Enabled Features: RxQ: 63 TxQ: 63 FdirHash

[    3.040850] ixgbe 0000:81:00.0 eth2: Intel(R) 10 Gigabit Network Connection

[    3.044725] ixgbe 0000:81:00.1 0000:81:00.1 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled

[    3.210429] ixgbe 0000:81:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0

[    3.260919] ixgbe 0000:81:00.1: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:80:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)

[    3.269678] ixgbe 0000:81:00.1 eth3: MAC: 3, PHY: 3, PBA No: 030C00-000

[    3.270207] ixgbe 0000:81:00.1: 0c:c4:7a:eb:35:a3

[    3.270500] ixgbe 0000:81:00.1 eth3: Enabled Features: RxQ: 63 TxQ: 63 FdirHash

[    3.276929] ixgbe 0000:81:00.1 eth3: Intel(R) 10 Gigabit Network Connection

[    3.292756] ixgbe 0000:0a:00.0 enp10s0f0: renamed from eth0

[    3.299150] ixgbe 0000:0a:00.1 enp10s0f1: renamed from eth1

[    3.333744] ixgbe 0000:81:00.0 enp129s0f0: renamed from eth2

[    3.347071] ixgbe 0000:81:00.1 enp129s0f1: renamed from eth3

[    9.388859] ixgbe 0000:0a:00.0: registered PHC device on enp10s0f0

[    9.779075] ixgbe 0000:0a:00.1: registered PHC device on enp10s0f1

[   10.185021] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0

[   10.555598] ixgbe 0000:81:00.1: registered PHC device on enp129s0f1

[   10.675605] ixgbe 0000:81:00.0 enp129s0f0: entered allmulticast mode

[   10.677409] ixgbe 0000:81:00.1 enp129s0f1: entered allmulticast mode

[   10.680855] ixgbe 0000:81:00.0 enp129s0f0: entered promiscuous mode

[   10.682723] ixgbe 0000:81:00.1 enp129s0f1: entered promiscuous mode

[   16.452049] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: None

[   16.836114] ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Up 10 Gbps, Flow Control: None

switch:

Code: 
cysco3850_s3#show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

        A - formed by Auto LAG


Number of channel-groups in use: 3
Number of aggregators:           3

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP        Te1/0/46(P)     Te1/0/48(P)     
2      Po2(SU)         LACP        Te1/0/42(P)     Te1/0/44(P)     
3      Po3(SD)         LACP        Te1/0/37(s)     Te1/0/40(s)


show etherchannel detail


Code: 
Group: 3 
----------
Group state = L2
Ports: 2   Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol:   LACP
Minimum Links: 0
          
          
                Ports in the group:
                -------------------
Port: Te1/0/37
------------
          
Port state    = Up Mstr Assoc In-Bndl
Channel group = 3           Mode = Active          Gcchange = -
Port-channel  = Po3         GC   =   -             Pseudo port-channel = Po3
Port index    = 0           Load = 0x00            Protocol =   LACP
          
Flags:  S - Device is sending Slow LACPDUs   F - Device is sending fast LACPDUs.
        A - Device is in active mode.        P - Device is in passive mode.
          
Local information:
                                LACP port    Admin     Oper    Port        Port
Port          Flags   State     Priority     Key       Key     Number      State
Te1/0/37      SA      bndl      32768        0x3       0x3     0x126       0x3D 
          
 Partner's information:
          
                     LACP port                      Admin  Oper   Port    Port
Port          Flags  Priority  Dev ID          Age  key    Key    Number  State
Te1/0/37      SA     255       0cc4.7aeb.35a2  13s  0x0    0xF    0x1     0x3D 
          
Age of the port in the current state: 0d:00h:02m:50s
          
Port: Te1/0/40
------------
          
Port state    = Up Mstr Assoc In-Bndl
Channel group = 3           Mode = Active          Gcchange = -
Port-channel  = Po3         GC   =   -             Pseudo port-channel = Po3
Port index    = 0           Load = 0x00            Protocol =   LACP
          
Flags:  S - Device is sending Slow LACPDUs   F - Device is sending fast LACPDUs.
        A - Device is in active mode.        P - Device is in passive mode.
          
Local information:
                                LACP port    Admin     Oper    Port        Port
Port          Flags   State     Priority     Key       Key     Number      State
Te1/0/40      SA      bndl      32768        0x3       0x3     0x129       0x3D 
          
 Partner's information:
          
                     LACP port                      Admin  Oper   Port    Port
Port          Flags  Priority  Dev ID          Age  key    Key    Number  State
Te1/0/40      SA     255       0cc4.7aeb.35a2  13s  0x0    0xF    0x2     0x3D 
          
Age of the port in the current state: 0d:00h:02m:49s
          
                Port-channels in the group:
                ---------------------------
          
Port-channel: Po3    (Primary Aggregator)
          
------------
          
Age of the Port-channel   = 3d:00h:40m:42s
Logical slot/port   = 12/3          Number of ports = 2
HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP
Port security       = Disabled
          
Ports in the Port-channel:
          
Index   Load   Port        EC state        No of bits
------+------+------+------------------+-----------
  0     00     Te1/0/37    Active             0
  0     00     Te1/0/40    Active             0
          
Time since last port bundled:    0d:00h:02m:49s    Te1/0/40
Time since last port Un-bundled: 0d:00h:04m:03s    Te1/0/40
 
_gabriel said:
are you running 3.16+ iperf3 version ? because it's CPU multi threads only from version 3.16, before it's only multi streams.
Click to expand...
root@S3ai:~# iperf3 -v
iperf 3.12 (cJSON 1.7.15)
Linux S3ai 6.8.12-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-1 (2024-08-05T16:17Z) x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment
root@S3ai:~#
 
I haven't exp with 10 Gb/s network but perhaps CPU limited with single stream connection, like inter Guests traffic over virtual bridge which is pure CPU.
try with iperf3 3.16 or newer version which can use CPU multi thread.
 
_gabriel said:
I haven't exp with 10 Gb/s network but perhaps CPU limited with single stream connection, like inter Guests traffic over virtual bridge which is pure CPU.
try with iperf3 3.16 or newer version which can use CPU multi thread.
Click to expand...
cpu is fine... around 20%
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom