LACP Problem with DELL R760 Server and 2 Network Cards

A

Alex78

New Member
Proxmox Subscriber
Feb 14, 2025
6
1
3
Hello,
we have a strange problem with the Proxmox server. We have DELL R760 servers with 4 x 25 GbE onboard cards (Broadcom) and an additional 2 x 25 GbE AddOn PCI card (Broadcom). A LACP Bond 802.3ad has been set up. The bond is stable. However, if we fetch data via this channel from a VM from the Internet or another data source, the connection can be slow at first, e.g. if you do an SCP or WGET. If you call the command several times, the connection is suddenly fast.

The bond consists of one port of the onboard card and one port of the add-on card. If we convert the bond and take both ports of the bond from one card, the problem does not occur. The connection is always fast.
Are there any known problems when building bonds with two different network cards?

Does everyone have any ideas?
 
We are using both integrated + addon network card for one LACP without problem on our Dell/Supermicro 10G setups.
 
PVE-Version? Kernel-Version? Maybe there are problems with the driver or LACP-Mode in switch is different from what is set in Proxmox?
 
Last edited:
we use the following config. Do you need further information?

Code: 
pveversion --verbose

proxmox-ve: 8.3.0 (running kernel: 6.8.12-8-pve)

pve-manager: 8.3.5 (running version: 8.3.5/dac3aa88bac3f300)

proxmox-kernel-helper: 8.1.1

proxmox-kernel-6.8: 6.8.12-8

proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8

proxmox-kernel-6.8.12-6-pve-signed: 6.8.12-6

proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4

ceph: 19.2.0-pve2

ceph-fuse: 19.2.0-pve2

corosync: 3.1.7-pve3

criu: 3.17.1-2+deb12u1

glusterfs-client: 10.3-5

ifupdown2: 3.2.0-1+pmx11

ksm-control-daemon: 1.5-1

libjs-extjs: 7.0.0-5

libknet1: 1.28-pve1

libproxmox-acme-perl: 1.5.1

libproxmox-backup-qemu0: 1.5.1

libproxmox-rs-perl: 0.3.4

libpve-access-control: 8.2.0

libpve-apiclient-perl: 3.3.2

libpve-cluster-api-perl: 8.0.10

libpve-cluster-perl: 8.0.10

libpve-common-perl: 8.2.9

libpve-guest-common-perl: 5.1.6

libpve-http-server-perl: 5.2.0

libpve-network-perl: 0.10.1

libpve-rs-perl: 0.9.1

libpve-storage-perl: 8.3.3

libspice-server1: 0.15.1-1

lvm2: 2.03.16-2

lxc-pve: 6.0.0-1

lxcfs: 6.0.0-pve2

novnc-pve: 1.5.0-1

proxmox-backup-client: 3.3.3-1

proxmox-backup-file-restore: 3.3.3-1

proxmox-firewall: 0.6.0

proxmox-kernel-helper: 8.1.1

proxmox-mail-forward: 0.3.1

proxmox-mini-journalreader: 1.4.0

proxmox-offline-mirror-helper: 0.6.7

proxmox-widget-toolkit: 4.3.6

pve-cluster: 8.0.10

pve-container: 5.2.4

pve-docs: 8.3.1

pve-edk2-firmware: 4.2023.08-4

pve-esxi-import-tools: 0.7.2

pve-firewall: 5.1.0

pve-firmware: 3.14-3

pve-ha-manager: 4.0.6

pve-i18n: 3.4.0

pve-qemu-kvm: 9.0.2-5

pve-xtermjs: 5.3.0-3

qemu-server: 8.3.8

smartmontools: 7.3-pve1

spiceterm: 3.3.0

swtpm: 0.8.0+pve1

vncterm: 1.8.0

zfsutils-linux: 2.2.7-pve1





Code: 
# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno12399np0
iface eno12399np0 inet manual
#Onboard 0

auto ens5f0np0
iface ens5f0np0 inet manual
#AdOnCard 0

auto eno12409np1
iface eno12409np1 inet manual
#Onboard 1

iface eno12419np2 inet manual
#Onboard 2

iface eno12429np3 inet manual
#Onboard 3

auto ens6f0np0
iface ens6f0np0 inet manual
#Ceph 100GbE 0

auto ens6f1np1
iface ens6f1np1 inet manual
#Ceph 100GbE 1

auto ens5f1np1
iface ens5f1np1 inet manual
#AdOnCard 1

auto bond0
iface bond0 inet manual
    bond-slaves eno12399np0 eno12409np1
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4
#Mgmt Bond

auto bond0.2152
iface bond0.2152 inet static
    address 10.173.17.10/24
    gateway 10.173.17.1
#VLAN 2152 - Proxmox Mgmt

auto bond1
iface bond1 inet manual
    bond-slaves ens5f0np0 ens5f1np1
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4
#VM Server Bond

auto bond2
iface bond2 inet static
    address 10.173.16.10/24
    bond-slaves ens6f0np0 ens6f1np1
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4
    mtu 9000
#Ceph Bond

auto bond0.718
iface bond0.718 inet static
    address 10.169.11.10/24
#Corosync1

auto bond1.719
iface bond1.719 inet static
    address 10.169.12.10/24
#Corosync2

auto vmbr1
iface vmbr1 inet manual
    bridge-ports bond1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#SDN Bridge

source /etc/network/interfaces.d/*[CODE]


[CODE]# lspci -nnk

00:00.0 System peripheral [0880]: Intel Corporation Ice Lake Memory Map/VT-d [8086:09a2] (rev 20)

    Subsystem: Dell Ice Lake Memory Map/VT-d [1028:0a6b]

00:00.1 System peripheral [0880]: Intel Corporation Ice Lake Mesh 2 PCIe [8086:09a4] (rev 20)

    Subsystem: Dell Ice Lake Mesh 2 PCIe [1028:0a6b]

00:00.2 System peripheral [0880]: Intel Corporation Ice Lake RAS [8086:09a3] (rev 20)

    Subsystem: Dell Ice Lake RAS [1028:0a6b]

00:00.4 Generic system peripheral [0807]: Intel Corporation Device [8086:0b23]

    Subsystem: Dell Device [1028:0a6b]

    Kernel driver in use: pcieport

00:0c.0 PCI bridge [0604]: Intel Corporation Device [8086:1bbc] (rev 11)

    DeviceName: PCH Root Port

    Subsystem: Dell Device [1028:0a6b]

    Kernel driver in use: pcieport

00:0e.0 PCI bridge [0604]: Intel Corporation Device [8086:1bbe] (rev 11)

    DeviceName: PCH Root Port

    Subsystem: Dell Device [1028:0a6b]

    Kernel driver in use: pcieport

00:14.0 USB controller [0c03]: Intel Corporation Emmitsburg (C740 Family) USB 3.2 Gen 1 xHCI Controller [8086:1bcd] (rev 11)

    Subsystem: Dell Emmitsburg (C740 Family) USB 3.2 Gen 1 xHCI Controller [1028:0a6b]

    Kernel driver in use: xhci_hcd

    Kernel modules: xhci_pci

00:14.2 RAM memory [0500]: Intel Corporation Device [8086:1bce] (rev 11)

    Subsystem: Dell Device [1028:0a6b]

00:14.4 Host bridge [0600]: Intel Corporation Device [8086:1bfe] (rev 11)

    Subsystem: Dell Device [1028:0a6b]

00:15.0 System peripheral [0880]: Intel Corporation Device [8086:1bff] (rev 11)

    Subsystem: Dell Device [1028:0a6b]

    Kernel driver in use: ismt_smbus

    Kernel modules: i2c_ismt

00:16.0 Communication controller [0780]: Intel Corporation Device [8086:1be0] (rev 11)

    Subsystem: Dell Device [1028:0a6b]

    Kernel modules: mei_me

00:18.0 SATA controller [0106]: Intel Corporation Sapphire Rapids SATA AHCI Controller [8086:1bf2] (rev 11)

    Subsystem: Dell Sapphire Rapids SATA AHCI Controller [1028:0a6b]

    Kernel driver in use: ahci

    Kernel modules: ahci

00:19.0 SATA controller [0106]: Intel Corporation Sapphire Rapids SATA AHCI Controller [8086:1bd2] (rev 11)

    Subsystem: Dell Sapphire Rapids SATA AHCI Controller [1028:0a6b]

    Kernel driver in use: ahci

    Kernel modules: ahci

00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:1b81] (rev 11)

    Subsystem: Dell Device [1028:0a6b]

00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:1bc9] (rev 11)

    Subsystem: Dell Device [1028:0a6b]

    Kernel driver in use: i801_smbus

    Kernel modules: i2c_i801

00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:1bca] (rev 11)

    Subsystem: Dell Device [1028:0a6b]

    Kernel driver in use: intel-spi

    Kernel modules: spi_intel_pci

02:00.0 PCI bridge [0604]: PLDA PCI Express Bridge [1556:be00] (rev 02)

03:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller [102b:0536] (rev 04)

    DeviceName: Embedded Video

    Subsystem: Dell Integrated Matrox G200eW3 Graphics Controller [1028:0a6b]

    Kernel driver in use: mgag200

    Kernel modules: mgag200

0c:00.0 System peripheral [0880]: Intel Corporation Ice Lake Memory Map/VT-d [8086:09a2] (rev 20)

    Subsystem: Intel Corporation Ice Lake Memory Map/VT-d [8086:0000]

0c:00.1 System peripheral [0880]: Intel Corporation Ice Lake Mesh 2 PCIe [8086:09a4] (rev 20)

    Subsystem: Intel Corporation Ice Lake Mesh 2 PCIe [8086:0000]

0c:00.2 System peripheral [0880]: Intel Corporation Ice Lake RAS [8086:09a3] (rev 20)

    Subsystem: Intel Corporation Ice Lake RAS [8086:0000]

0c:00.4 Generic system peripheral [0807]: Intel Corporation Device [8086:0b23]

    Subsystem: Intel Corporation Device [8086:0000]

    Kernel driver in use: pcieport

21:00.0 System peripheral [0880]: Intel Corporation Ice Lake Memory Map/VT-d [8086:09a2] (rev 20)

    Subsystem: Intel Corporation Ice Lake Memory Map/VT-d [8086:0000]

21:00.1 System peripheral [0880]: Intel Corporation Ice Lake Mesh 2 PCIe [8086:09a4] (rev 20)

    Subsystem: Intel Corporation Ice Lake Mesh 2 PCIe [8086:0000]

21:00.2 System peripheral [0880]: Intel Corporation Ice Lake RAS [8086:09a3] (rev 20)

    Subsystem: Intel Corporation Ice Lake RAS [8086:0000]

21:00.4 Generic system peripheral [0807]: Intel Corporation Device [8086:0b23]

    Subsystem: Intel Corporation Device [8086:0000]

    Kernel driver in use: pcieport

21:01.0 PCI bridge [0604]: Intel Corporation Device [8086:352a] (rev 30)

    DeviceName: Root Port

    Subsystem: Intel Corporation Device [8086:0000]

    Kernel driver in use: pcieport

22:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1751] (rev 12)

    DeviceName: Integrated NIC 1 Port 1-1

    Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E BCM57504 4x25G OCP3.0 [14e4:5045]

    Kernel driver in use: bnxt_en

    Kernel modules: bnxt_en

22:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1751] (rev 12)

    DeviceName: Integrated NIC 1 Port 2-1

    Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E BCM57504 4x25G OCP3.0 [14e4:5045]

    Kernel driver in use: bnxt_en

    Kernel modules: bnxt_en

22:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1751] (rev 12)

    DeviceName: Integrated NIC 1 Port 3-1

    Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E BCM57504 4x25G OCP3.0 [14e4:5045]

    Kernel driver in use: bnxt_en

    Kernel modules: bnxt_en

22:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1751] (rev 12)

    DeviceName: Integrated NIC 1 Port 4-1

    Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E BCM57504 4x25G OCP3.0 [14e4:5045]

    Kernel driver in use: bnxt_en

    Kernel modules: bnxt_en

8a:01.0 PCI bridge [0604]: Intel Corporation Device [8086:352a] (rev 30)

    DeviceName: SLOT 6

    Subsystem: Intel Corporation Device [8086:0000]

    Kernel driver in use: pcieport

8b:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1750] (rev 11)

    Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E P2100D BCM57508 2x100G QSFP PCIE [14e4:d124]

    Kernel driver in use: bnxt_en

    Kernel modules: bnxt_en

8b:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1750] (rev 11)

    Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E P2100D BCM57508 2x100G QSFP PCIE [14e4:d124]

    Kernel driver in use: bnxt_en

    Kernel modules: bnxt_en

9f:00.0 System peripheral [0880]: Intel Corporation Ice Lake Memory Map/VT-d [8086:09a2] (rev 20)

    Subsystem: Intel Corporation Ice Lake Memory Map/VT-d [8086:0000]

9f:00.1 System peripheral [0880]: Intel Corporation Ice Lake Mesh 2 PCIe [8086:09a4] (rev 20)

    Subsystem: Intel Corporation Ice Lake Mesh 2 PCIe [8086:0000]

9f:00.2 System peripheral [0880]: Intel Corporation Ice Lake RAS [8086:09a3] (rev 20)

    Subsystem: Intel Corporation Ice Lake RAS [8086:0000]

9f:00.4 Generic system peripheral [0807]: Intel Corporation Device [8086:0b23]

    Subsystem: Intel Corporation Device [8086:0000]

    Kernel driver in use: pcieport

9f:01.0 PCI bridge [0604]: Intel Corporation Device [8086:352a] (rev 30)

    DeviceName: SLOT 5

    Subsystem: Intel Corporation Device [8086:0000]

    Kernel driver in use: pcieport

a0:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] (rev 01)

    Subsystem: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:4141]

    Kernel driver in use: bnxt_en

    Kernel modules: bnxt_en

a0:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] (rev 01)

    Subsystem: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:4141]

    Kernel driver in use: bnxt_en

    Kernel modules: bnxt_en
 
Normally that should work. What about the firmware versions of the cards? Did u update them?
 
I guess when you bond two ports from different NIC card, them has been separated to two CPU socket, may you can check both card's PCIe slot belongs to which CPU socket.
 
What about error rates, what about cabling issues, what about iperf, does the issue go away if you use 2 ports on the NIC, are you disabling or mismatching any offloading across the bonds, how is the switch set up, does it have the same LACP rates, do you see any errors on the switch or on the kernel (dmesg) or in the system logs.
 
Last edited:
Switch and Cables have no errors. The issue is not existent, when I use a bond with one NIC. If I use a bond with 2 Network Cards, then I get this behavier, that the traffic is sometimes very slow. LACP rate shows 50.000. I have checked the dmesg and I get some errors:
Code: 
[   85.813013] tap108i0: entered allmulticast mode
[   85.813067] fwbr108i0: port 2(tap108i0) entered blocking state
[   85.813068] fwbr108i0: port 2(tap108i0) entered forwarding state
[  224.940192] tap108i0: left allmulticast mode
[  224.940226] fwbr108i0: port 2(tap108i0) entered disabled state
[  224.953085] fwbr108i0: port 1(fwln108i0) entered disabled state
[  224.953236] VLAN2147: port 2(fwpr108p0) entered disabled state
[  224.953359] fwln108i0 (unregistering): left allmulticast mode
[  224.953362] fwln108i0 (unregistering): left promiscuous mode
[  224.953363] fwbr108i0: port 1(fwln108i0) entered disabled state
[  224.961024] fwpr108p0 (unregistering): left allmulticast mode
[  224.961026] fwpr108p0 (unregistering): left promiscuous mode
[  224.961027] VLAN2147: port 2(fwpr108p0) entered disabled state

I am not shure, if I have a constelation in the network setup in Proxmox, which is not the best:

1742504484347.png
 
First, LACP rate of 50 is unusual, it should be 30 (slow) or 1 (fast). What is the switch config actually.
bond-xmit-hash-policy layer3+4 should be layer2 or 2+3 to be standards compliant.

Second, your cards are capable of offloading LACP, off course they can't do that when they aren't the same NIC, make sure to disable offloading.

The other thing I can think about is that LACP picks a MAC address from one of the links and makes that the MAC address of the NIC, not sure if that propagates across the other ports (eg. B0:20, B0:21 ...) when the trunk is up.
 
You must log in or register to reply here.
Top Bottom