Interface not picking up traffic without promiscous mode

[Sebastian Schubert]

Hi folks,

i've a very weird behaviour with a cluster i built recently. The nics only pic up traffic after setting them manually to promiscous mode. Anyone ever run into this problem?

root@proxmox001:~# lspci -s 82:00.0 -v
82:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
    Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT
    Flags: bus master, fast devsel, latency 0, IRQ 67, NUMA node 1, IOMMU group 4
    Memory at 3c000000000 (64-bit, prefetchable) [size=32M]
    Expansion ROM at cc100000 [disabled] [size=1M]
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [48] Vital Product Data
    Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
    Capabilities: [c0] Vendor Specific Information: Len=18 <?>
    Capabilities: [40] Power Management version 3
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1c0] Secondary PCI Express
    Capabilities: [230] Access Control Services
    Kernel driver in use: mlx5_core
    Kernel modules: mlx5_core

 

[VictorSTS]

Try updating the firmware [1]. I've had funky behavior on Mellanox NIC's when using some firmware versions (can't recall which ones) and the inbox driver (the one provided with the Linux/PVE kernel).

Also, these NIC use VLAN offload but can only offload a few VLAN's (512 for the ConnectX-4 IIRC) and have funky behavior if you try to use more via the bridge-vids 2-4092 directive in /etc/network/interfaces. Try also to disable VLAN offloading, i.e.:

[...]

auto vmbr0
iface vmbr0 inet static
        address xx.xx.xx.xx/24
        gateway 10.43.4.254
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        pre-up /usr/sbin/ethtool -K ens3f0np0 rx-vlan-filter off
        pre-up /usr/sbin/ethtool -K ens3f1np1 rx-vlan-filter off

[...]

[1] https://www.mellanox.com/support/firmware/mlxup-mft