Mellanox Connectx-3 Cards and Proxmox8

[T20A02]

Good afternoon, all,
I have recently upgraded to proxmox 8, on ym three IBM iDataPlex dx360 m4's. I have been attempting to configure a 10gb connection via SFP from the Mellanox Connectx cards to my unify 16 port SFP switch. I have been successful getting a link light between the two, but I have been having trouble getting the Mellanox cards to behave properly with with Debian 12. It seems the last supported version i can find drivers for is for Debian 11.3 and as far as i know they are EOL, and therefor will not release drivers for the platform. Anyone have any ideas on how to proceed? I have been betting on using the system, but i don't want to downgrade versions. Any generic drivers out there?

Thanks

 

[T20A02]

Alright, solved my issue, I was just foolish, I had to switch the specific cards in this system from their inifiniband mode, to standard Ethernet mode. Once I swapped that, and manually upped the interfaces, I did not have any further issues.

 

[RexInTheCity]

Alright, solved my issue, I was just foolish, I had to switch the specific cards in this system from their inifiniband mode, to standard Ethernet mode. Once I swapped that, and manually upped the interfaces, I did not have any further issues.

Do you have any specifics on how you did that? I'm new to the Mellanox platform and haven't been able to get my cards working in Proxmox.

 

[Djjvb]

Me too! I am also new. The ports show, but when I started a VM for OpnSense it got stuck in a loop with the error: mlx_core0: Unable to determine PCI device chain minimum BW. I found a post it had something to do with the infiniband mode? So I want to try that.

 

[salmawisokys]

Upgrading to a new version of an operating system can sometimes introduce compatibility issues, especially when it comes to hardware drivers. In your case, it seems that the Mellanox ConnectX cards are not behaving properly with Debian 12, and the last supported drivers you found are for Debian 11.3.

 

[rigorm]

hopefully, this wont be the case, I just got 4x ConnectX-3 MCX312-XCBT on the way to replace some weird behaving HP NC522SFP in my cluster. I'm in the middle of upgrading my homelab to 8.1.4 and some card just died while upgrading (was fine in 7.4.17 but never came up when done at 8.1.4).

 

[Djjvb]

I don't think it's the case. I've seen posts of people which were able to get sr-iov working for this card under proxmox 8? That to mee seems like it is working. I've found out that it only occurs when I do pci passthrough of the whole card. As said they are visible in proxmox. So when I make bridges, they show up in opnsense as 10GBase-T. Only one port is active, the other is down. So still have to find out why that is. But I don't think in my case it is/was the ethernet mode.

 

[Dunuin]

Upgrading to a new version of an operating system can sometimes introduce compatibility issues, especially when it comes to hardware drivers. In your case, it seems that the Mellanox ConnectX cards are not behaving properly with Debian 12, and the last supported drivers you found are for Debian 11.3.

Those ConnectX-3 are EoL with dropped software support.

 

[T20A02]

I saw the same thing about the software support, but the actual drivers for them are still built in. I don't know which server you are using but there was a config file/ firmware switch, as the card's in my case could be used for infiniband and they needed to be switched to sfp+, and now im using them with qsfp to sfp+ adapters.

 

[alexskysilk]

Those ConnectX-3 are EoL with dropped software support.

source? if you mean ofed inbox drivers, they were never required for functionality. I still have plenty of nodes running connectx3 nics on proxmox 8 without issue.