Mellanox ConnectX-4 LX and brigde-vlan-aware on Proxmox 8.0.1

joxx75

New Member
Nov 6, 2022
3
3
3
Hi everyone! This is my first post here so bear with me. I hope I don't break too many rules but I didn't find any guidelines.

I've been looking for a solution to enable bridge-vlan-aware on Mellanox ConnectX-4 LX (MCX4121A-ACAT, firmware 14.32.1010) on Proxmox 8.0.1 (6.2.16-4-pve) with the inbox driver. With bridge-vlan-aware enabled I didn't receive any network traffic. Didn't test the Mellanox OFED driver but this kernel isn't supported yet by it anyway. With vlan-aware enabled I could see traffic being sent out correctly and reaching my switch but traffic didn't get back unless the interface was set to promiscuous mode, leading me to look into configuration of the Mellanox card if any of the many many options caused traffic to be filtered. I tried toggling some possibly relevant options available via ethtool and via mlxconfig with no success.

Reading the mlx5 driver documentation I came across bridge offloading that mentioned eswitch and switchdev mode. https://docs.kernel.org/next/networking/device_drivers/ethernet/mellanox/mlx5.html#bridge-offload

Checking my system I saw that eswitch on my Mellanox card was set to legacy mode, and I tested to set it to switchdev, and traffic started to flow.
Code:
# devlink dev eswitch show pci/0000:01:00.0
pci/0000:01:00.0: mode legacy inline-mode none encap-mode basic
# devlink dev eswitch set pci/0000:01:00.0 mode switchdev
# devlink dev eswitch show pci/0000:01:00.0
pci/0000:01:00.0: mode switchdev inline-mode link encap-mode basic

I don't know why this is, this is too advanced for me.

Looking to enable this at boot time I came across this Intel document (I don't use SR-IOV VFs though so that part isn't relevant)
https://edc.intel.com/content/www/u...itchdev-mode-with-linux-bridge-configuration/

They enabled it first after the bridge was created and the physical ports was added, which I guess corresponds to after the bridge interface has been brought up. I tried doing it earlier than this with no success.

I currently have things working at boot with this config in /etc/network/interfaces. Verified on a second host with an identical card that I hadn't messed around with during testing so I don't think I've changed anything else.

Code:
auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp1s0f0np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 10-520
        # 0000:01:00.0 corresponds to enp1s0f0np0
        post-up devlink dev eswitch set pci/0000:01:00.0 mode switchdev

auto vlan20
iface vlan20 inet static
        address 192.168.20.3/24
        vlan-raw-device vmbr0

I needed vlan-aware since in my homelab I wanted to try running Gluster/CTDB/Samba directly on the hosts and my VM's needed to be able to communicate with the CTDB public ips on the same vlan. Without bridge-vlan-aware enabled I couldn't have any VM with a nic attached to the same vlan (vmbr0 vlan 20) that CTDB/Samba on the host used, it broke the networking.

I hope this will help someone else stuck in my situation, or perhaps having a better fix for this problem than a post-up command.
 
@joxx75 i've just update FW (latest is from year 2020) on my mellanox connectx-4 416-bcat and now Vlan aware bridge is working as it should (traffic is passing).
We have cards from different families and I use the newest available firmware for my card. I haven't looked further into this or tried with newer software yet in case the problem with legacy mode has been resolved.
 
We have cards from different families and I use the newest available firmware for my card. I haven't looked further into this or tried with newer software yet in case the problem with legacy mode has been resolved.
By different family you mean ConnectX-4 LX and ConnectX-4 EN ?
 
Thank you for this post! Switching to "switchdev" worked. I got Few varieties of Mellanox Connectx4/ConnectX4Lx and all of them having this issue. I'm on the latest firmware on all of my cards and on Proxmox 8.
 
Thanks for this working fix :)

I've been using my ConnectX-4 LX for a while just never with VLANs

And to my dismay i couldn't get it to work :(
 
It seems like this is also the 'fix' to my issue.

My configuration includes a bridge which has a 'stacked' vlan interface for cluster communication. I was unable to get any traffic flowing over this vlan device unless a vm was running on the 'lower' bridge or I ran tcpdump on the bridge / manually set it to promiscuous mode.

Configuring the physical interface without any vlan or bridge in between works without any modification.

I am using a Mellanox ConnectX-5 (MCX512A-ACAT-ML). My configuration is something like this:

Code:
[...]
auto enp1s0f0np0
iface enp1s0f0np0 inet manual
# MCX512A-ACAT-ML P1

auto enp1s0f1np1
iface enp1s0f1np1 inet manual
# MCX512A-ACAT-ML P2

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp1s0f0np0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 33,34,35
    post-up devlink dev eswitch set pci/0000:01:00.0 mode switchdev

auto vmbr0.33
iface vmbr0.33 inet static
    address 10.33.100.10/24
#CLUSTER
[...]

Bash:
mstflint -d 01:00.0 q
Image type:            FS4
FW Version:            16.35.3502
FW Release Date:       27.12.2023
Product Version:       16.35.3502

Thanks @joxx75 !

If someone can help explain 'why' and 'if' this is the correct way I'd be pleased to learn. The internet so far has not much information about this issue rather than 'install some proprietary NVIDIA driver, update your cards fw, it might fix it'[...].
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!