Connectivity issue with Mellanox ConnectX-3 NIC

[royalj7]

Hello folks. I'm relatively new to Proxmox and recently jumped into 10GB networking. I picked up a Mellanox MCX311A-XCAT SPF+ NIC from Ebay and installed it in my server. The home server is literally what I had laying around in spare parts, and consists of a i5-4570S in a MSI B85M-E45 motherboard. I have Proxmox 6.4 installed and since I'm running the box headless, I stuck the ConnectX-3 in the 3x16 slot (only one that would work, as it only has two 2x1 slots and a PCI slot).

The problem I'm having is when I install the Mellanox NIC, it seems to disable the onboard 1GB connection. I lose all remote access to the machine and when I connect a monitor and keyboard locally, I can't ping anything on my LAN. “ip a” lists an IP for my bridge interface, vmbr0, but my router doesn’t show it as a client when the ConnectX-3 is installed. When I unplug the Mellanox, everything goes back to normal, no problems with remote or internet access to or from the box. Problem is very repeatable.

The fact that I don't have any connectivity when I install the ConnectX-3 makes troubleshooting pretty difficult...everything is sneaker net at that point. There is no reason I shouldn't be able to use the 1GB ethernet NIC when the Mellanox is installed is there? As I'm working to get the Mellanox configured (which will be another trial I'm sure) I'd like to have the onboard NIC working. I'm pretty much a Linux newbie, so I think there is probably something basic that I'm missing. Looking for any suggestions.

Thanks!

 

[mira]

The interfaces get renamed probably when you add the additional NIC.
Can you provide the output of ip l when the additional NIC is installed? And cat /etc/network/interfaces when you have network.

 

[Dunuin]

I think mira is right.

I'm also running the same NIC and it worked fine with PVE6 and still works with PVE7 which is based on Debian 11.0.

But I'm not sure how futureproof this NIC is. I heard the Connectx-3 support stopped with drivers for Debian 10.8.

 

[royalj7]

Thanks folks. I'm not at the sever right now (blasted work blocks other VPN connections), but will post the information this evening.

On a related note, I heard the same things about the ConnectX-3 driver support. Anyone have suggestions on reasonable cost 10GB NICs for homelab use that might be alittle newer?

 

[spirit]

maybe it could help :

with connect-x3 and some old servers, I need to use " pci=realloc=off" in grub options to get it work.

 

[royalj7]

I've had a chance to work on this some more, and as mira alluded to, adding the Mellanox NIC caused my interfaces to be renamed, and the virtual bridge, vmbr0, was bound to an interface that no longer existed. My 1Gb integrated NIC was named enp2s0. After I added the ConnectX-3, a ip-l showed NICs enp1s0 and enp3s0. I'm sure there is a good reason Linux handles the NICs this way, but is pretty unintuitive coming from a Windows environment.

After adding the Mellanox in, I hooked up locally to the box and edited the /etc/network/interfaces file to reference enp3s0, which got me back my 1Gb NIC. Now I just have to find out how to add the 10Gb NIC in the same subnet but different IP address. Wish me luck!

 

[Dunuin]

I've had a chance to work on this some more, and as mira alluded to, adding the Mellanox NIC caused my interfaces to be renamed, and the virtual bridge, vmbr0, was bound to an interface that no longer existed. My 1GB integrated NIC was named enp2s0. After I added the ConnectX-3, a ip-l showed NICs enp1s0 and enp3s0. I'm sure there is a good reason Linux handles the NICs this way, but is pretty unintuitive coming from a Windows environment.

Because the enumeration is based on the hardware. That way it only changes when you change your hardware. Before it was just random names that might change without changing anything.

After adding the Mellanox in, I hooked up locally to the box and edited the /etc/network/interfaces file to reference enp3s0, which got me back my 1GB NIC. Now I just have to find out how to add the 10GB NIC in the same subnet but different IP address. Wish me luck!

That is something you shouldn't do in general. If you want to send a packet to any host in your network, how should the host know if it should use the slow 1 Gbit NIC or the fast 10 Gbit NIC if both NICs are in the same network? Maybe it is then always choosing the slow 1 GBit NIC and your 10 Gbit NIC isn't used at all.