Mellanox ConnectX-4 LX and brigde-vlan-aware on Proxmox 8.0.1

We can also confirm that switchdev is the answer to this.

You can also setup a Systemd Service to make sure it works everytime or on reboots.

create a new systemd service file:
nano /etc/systemd/system/switchdev.service

Put in:

#############START FILE CONTENTS BELOW#################
[Unit]
Description=Enable SwitchDev mode for Mellanox NICs
After=network.target

[Service]
ExecStart=/usr/sbin/devlink dev eswitch set pci/0000:41:00.0 mode switchdev
ExecStart=/usr/sbin/devlink dev eswitch set pci/0000:41:00.1 mode switchdev
ExecStart=/usr/sbin/devlink dev eswitch set pci/0000:81:00.0 mode switchdev
ExecStart=/usr/sbin/devlink dev eswitch set pci/0000:81:00.1 mode switchdev
ExecStart=/usr/sbin/devlink dev eswitch set pci/0000:01:00.0 mode switchdev
ExecStart=/usr/sbin/devlink dev eswitch set pci/0000:01:00.1 mode switchdev
RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=multi-user.target

#############END FILE CONTENTS ABOVE#################

Just double check the correct devlink execution path with: which devlink or command -v devlink

You can also check the correct NIC ids to put into the file with: devlink dev show

Then save the file, and then run these commands:
systemctl daemon-reload
systemctl enable switchdev.service
systemctl start switchdev.service
systemctl status switchdev.service

When no errors then reboot

Thank you so much for this work-around @joxx75 it really saved us a lot of stress and time :)
 
I was trying to follow along with the above installing the debian 12.1 drivers..... and I am getting this message and unsure how to continue.
Can you describe your procedure? ty

RC


##################

This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements...
Removing old packages...

Error: One or more packages depends on MLNX_OFED_LINUX.
Those packages should be removed before uninstalling MLNX_OFED_LINUX:

librados2 qemu-server libcephfs2 libradosstriper1 libiscsi7 mft-autocomplete cockpit-benchmark libpve-guest-common-perl python3-rados pve-container librgw2 librados2-perl ceph-fuse pve-manager proxmox-ve ceph-common fio libpve-storage-perl python3-rbd python3-rgw spiceterm python3-cephfs pve-ha-manager pve-qemu-kvm librbd1

To force uninstallation use '--force' flag.
 
BillyTheKid said:
With FW Version 14.32.1900 (FW Release Date: 25.8.2024) my MCX4121A-ACAT works out of the box on proxmox 8.3.2 (bridge-vlan-aware yes).
Click to expand...
I updated my card (HP 817753-B21 using ConnectX-4 Lx) to 14.32.1900, but unfortunately the problem remains (Proxmox 8.4).

Using the devlink dev eswitch set pci/0000:01:00.0 mode switchdev workaround works, but it'd be great if this would be supported natively. Is there some issue opened somewhere for this (not on this forum, as I assume devs will not track our messages here)?

Related posts: https://forum.proxmox.com/threads/no-vlan-connection-after-reboot.138548/.

Also, I got some errors like:

mlx5e_vport_context_update_vlans:186(pid: 1995): netdev vlans list size (4095) > (512) max vport list size, some vlans will be dropped

But those seem related to a different issue.
 
@Held

I have no idea why this works for me. Perhaps because I have a different setup with multiple NICs?
Here's my configuration:
The Mellanox card serves as the LAN port of a VM Opnsense (enp6mlxLAN). The Proxmox host is connected to an Intel NIC (enp4intPVE), so I can always access the Proxmox host via an alternative router if problems arise. Attached is my /etc/network/interfaces file.

Good luck
 

Attachments

BillyTheKid said:
@Held

I have no idea why this works for me. Perhaps because I have a different setup with multiple NICs?
Here's my configuration:
The Mellanox card serves as the LAN port of a VM Opnsense (enp6mlxLAN). The Proxmox host is connected to an Intel NIC (enp4intPVE), so I can always access the Proxmox host via an alternative router if problems arise. Attached is my /etc/network/interfaces file.

Good luck
Click to expand...
Thanks for sharing your config. I noticed you're doing a pre-up ethtool -G enp1s0f0np0 tx 2048 rx 2048 on the device, so for a test I've executed the same on my adapter (after changing mode from switchdev to legacy again), but the problem remains.
Apart from this, my config is pretty similar to yours.

Code: 
# 10G link (bottom SFP+/SFP28)
auto enp1s0f0np0
iface enp1s0f0np0 inet manual
auto vmbr1
iface vmbr1 inet manual
    bridge-ports enp1s0f0np0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    # Max 512 for netdev vlan list size vs. max vport list size
    bridge-vids 2-500
    # Workaround for Mellanox issue - see https://forum.proxmox.com/threads/mellanox-connectx-4-lx-and-brigde-vlan-aware-on-proxmox-8-0-1.130902/
    post-up devlink dev eswitch set pci/0000:01:00.0 mode switchdev
 
Very strange that it works for me.
To test this, I connected another VM to the same bridge I use for the OPNsense(VM) - LAN interface (I use a 10G SFP+ module instead of SFP28 in my MCX4121A-ACAT), where all of my local network's VLANs exist: This test-vm works perfectly, regardless of the assigned VLAN tag. So, the "VLAN-aware" option of the bridge seems to be working.

Previously, the option was actually not necessary because I only used VLANs in the OPNsense VM on the same 10G interface.
 
I’m on PVE 9.0.10 and just swapped a PCIe gen2.0 Intel card for a PCIe gen3.0 Mellanox ConnectX-4 LX (MCX4121A-ACUT) to make better use of the limited lanes on my mother board’s slot. The MCX4121A-ACUT is running the latest F/W 14.32.1900. My PVE host is unable to pass traffic on the VLAN aware bridge (i.e., via the default, untagged VLAN present on the bridge) without adding the eswitch switchdev setting per other posts in this thread. I also needed to limit the number of bridge-vids as noted here too. I haven’t tested communications to/from VM’s using the tagged VLANs yet though. This all worked out of the box with the Intel card.

This is my current configuration for the bridge:
Code: 
iface enp6s0f0np0 inet manual

# Workaround 2025-09-18: The Mellanox ConnectX-4 LX NIC requires mode
# switchdev to be set below in order to function with a VLAN-aware bridge.
# Additionally no more than 512 bridge-vids can be specified due to an apparent
# Mellanox driver limitation. Thus we specify only "in-use" VLANs rather than the
# default range (2-4094) that the Proxmox GUI typically adds when enabling
# vlan-aware on a bridge interface.
# See https://forum.proxmox.com/threads/mellanox-connectx-4-lx-and-brigde-vlan-aware-on-proxmox-8-0-1.130902/

auto vmbr0
iface vmbr0 inet static
        address 192.168.2.80/24
        gateway 192.168.2.254
        bridge-ports enp6s0f0np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 10 20
        post-up devlink dev eswitch set pci/0000:06:00.0 mode switchdev

Even with the config above, I am seeing occasional messages in syslog from mlx5_core as follows:

Code: 
Sep 19 02:43:06 pve1 kernel: mlx5_core 0000:06:00.0: E-Switch: Failed to lookup bridge port to add MDB (MAC=xx:xx:xx:xx:xx:xx,vport=65535)
Sep 19 02:43:06 pve1 kernel: mlx5_core 0000:06:00.0 enp6s0f0np0: Failed to add Port Multicast Database entry (object id=2) with error: -EINVAL (-22).
Failure in updating the port's Multicast Database could lead to
multicast forwarding issues.

EDIT: Here’s an updated configuration including workarounds to address the MDB related errors. Since the workarounds likely result in some performance penalty, I plan to retest without the workarounds whenever a new mlx5_core driver and/or MCX4121A-ACUT F/W is released.

Code: 
iface enp6s0f0np0 inet manual
        pre-up ethtool -K $IFACE rx-vlan-offload off
        pre-up ethtool -K $IFACE rx-vlan-filter off

# Workarounds 2025-09-19: The Mellanox ConnectX-4 LX NIC requires mode
# switchdev to be set below in order for the VLAN-aware bridge to pass traffic.
# Additionally no more than 512 bridge-vids can be specified on the bridge due to a
# Mellanox driver limitation. Thus we specify only "in-use" bridge-vids rather than
# the default range (2-4094) typically added when setting a bridge vlan aware in the
# Proxmox GUI. In addition, this network configuration contains two pre-up commands
# under enp6s0f0np0 to disable rx vlan offload and filtering. By default these
# were requested on by the driver, but remained off nonetheless (as confirmed using
# ethtool). Setting these off explicitly as well as disabling multicast snooping on
# vmbr0 were both recommended fixes / workarounds to eliminate MDB related errors
# reported by the mlx5_core driver and noted in my post to the proxmox forun thread
# below. Ideally to improve performance, vlan offloading and filtering should remain
# on and multicast_snooping enabled. The need for these workarounds should thus be
# revisited when newer versions of the mlx5_core driver and/or MCX4121A-ACUT firmware
# is released.
# https://forum.proxmox.com/threads/mellanox-connectx-4-lx-and-brigde-vlan-aware-on-proxmox-8-0-1.130902/

auto vmbr0
iface vmbr0 inet static
        address 192.168.2.80/24
        gateway 192.168.2.254
        bridge-ports enp6s0f0np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 10 20
        post-up devlink dev eswitch set pci/0000:06:00.0 mode switchdev
        post-up echo 0 > /sys/class/net/vmbr0/bridge/multicast_snooping
 
Last edited:
You must log in or register to reply here.
Top Bottom