Network connectivity issue when vmbr interface is present in interfaces file and Mellanox ConnectX-5

[JustinG]

Hello,

I've been fighting an issue for about a week now with some strange networking-related issues. My current setup is as follows:

3x brand-new ProxMox VE 7.3-6 hosts installed on Dell R6515 servers running kernel 5.15.85-1-pve. My /etc/network/interfaces file looks like the following:

Our management network is on VLAN 79, and our CEPH network is on VLAN 77. With the current above configuration, upon reboot, our hypervisors and VMs have no connectivity whatsoever, including to other hosts in the same layer-2 network (incomplete ARP entries are present, and all pings show "Destination host unreachable"), despite "ip a" showing all interfaces as UP:




Running tcpdump on both the enp65s0f1np1.79 interface as well as vmbr1 only shows ARP traffic, nothing else. The only way to fix this issue is to comment out both vmbr interfaces from /etc/network/interfaces, reboot the node entirely (systemctl restart networking alone doesn't have any effect), then uncomment the vmbr interfaces and run systemctl restart networking. After all these steps, everything is up and running again without issue.

I have also attempted assigning the management network 79 to vmbr1.79 instead of using the raw bridge device, but the same exact issue occurs.

I should mention this issue is only occuring on our ProxMox nodes that have Mellanox ConnectX-5s installed, we have other hosts utilizing Broadcom NICs that don't exhibit this same behavior. I already ensured our ConnectX5 NICs have been updated to the latest available firmware.



I am running out of ideas at this point unfortunately, short of swapping out our physical NIC hardware. Has anyone experienced similar issues in the past?

 

[ev5unleash]

Hi Justin,

Came here Googling a similar issue I seem to be having as well. I am able to pass traffic through Mellanox ConnectX5 cards at the host level, but no traffic passes to the VMs. ARPs seem to be going through to other devices on the network, but nothing else passes through. I have noticed that the VMs get a NIC error when trying to pass traffic on the virtIO interface.

Here's what I have tried so far

1. Trying the identical configurations with/without a basic linux bridge on an Intel 10G card and traffic passes normally.
2. I tried a different brand of 40G switches and transceivers.
3. I flashed the latest Mellanox firmware and installed the newest Debian drivers from the website.

I'm going to fully test Windows Server just to make sure nothing odd is occurring on the Mellanox cards themselves, but so far, everything seems to be pointing to a Proxmox issue.

Hope we can find a resolution here.

Thanks,

 

[JustinG]

If it matters at all, we were forced to switch hardware to Intel NICs for our original issue, which was resolved as soon as we changed NIC vendors.

I'm experiencing this issue yet again on another host with a Mellanox ConnectX-5. This is concerning as we have quite a few physical hosts that utilize these cards.

If we could be of any assistance in providing more information we'd be happy to do so, just need to know where to look.

Thanks.