Making SR-IOV work for a Mellanox ConnectX-3 (single port, non-pro) NIC

[kevindd992002]

So I need to make SR-IOV work for one of my NICs (Mellanox ConnectX-3 non-pro) in my Dell Optiplex 5050 system, so it can be shared by two VM's, opnsense and debian. These are the two articles that I'm trying to work with:

https://enterprise-support.nvidia.c...re-sr-iov-for-connectx-3-with-kvm--ethernet-x
https://forum.proxmox.com/threads/h...ox-connectx-3-cards-for-sriov-and-vfs.121927/

I have a couple of questions:

1. Do I need to install the OFED driver in proxmox? The 2nd link says it isn't compatible with proxmox but that's because she was using proxmox 7.3 that time. I read somewhere that it is now supported in 8.1.

2. Are all the steps in the official Nvidia article applicable for proxmox 8.1? I'm inclined to follow that guide, of course, instead of the one in the 2nd link.

3. After enabling SR-IOV on the OFED driver, the 2nd link says to perform either of these:

--OPTION 1: I havent tested i was so annoyed that once I got it working I wasn't in the mood to undo and redo--
--All you may have to do next is--
update-initramfs -u

--OPTION 2: If that doesn't work then go with the following--
modprobe -r mlx4_en mlx4_ib
modprobe mlx4_en
update-initramfs -u

While the Nvidia article simply says to restart the driver. Which is which?

4. Will I be able to install the OFED driver on the opnsense VM?

Sorry, I'm a proxmox/Linux noob so bear with me. For context, I have pcie passthrough configured on the Dell BCM57810S NIC for the opnsense VM. Pcie passthrough is used because only the opnsense VM will need both ports of this NIC for 2 x 2.5Gb WAN connections. No need to share it with other VM's.

 

[Dunuin]

Are you able to enable SR-IOV support in the NICs firmware?
Last time I tried that there was an error in the firmware so it couldn't save my changes. I googled a lot and general census was that the firmware is crap and you have to do some hacky stuff to manually write your own configs.
Don't wanted to brick my NIC by flashing some customized firewares so I dropped the idea of SR-IOV.

I'm using three MCX311A-XCATs. Also keep in mind that the NIC is EoL and won't receive any new drivers/firmwares. I think latest official supported Debian was 10 or 11. So in case you need to use some of the Connextx-3 software for updating the firmware or configuring stuff you might need to boot an older OS as PVE8 is based on the unsupported Debian 12.

 

[kevindd992002]

I haven't even tried yet because I wanted to understand the whole process first before diving in. From what I understand though, you can use the mstflint package to get/set SR-IOV on the NIC's firmware. That's also one of the instructions in that 2nd link. So I don't understand why would installing the MLNX_OFED driver be one of the prerequisites listed in the Nvidia article?

 

[kevindd992002]

Looks like the new MLNX_OFED driver does support Debian 12:

https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

 

[kevindd992002]

@Dunuin is this where you were stuck before when you were trying to set the config on the NIC's firmware:

https://forums.servethehome.com/ind...n-proxmox-for-mellanox-cx3.42618/#post-404650

 

[kevindd992002]

And here's my whole thread about it:

https://forums.servethehome.com/ind...n-proxmox-for-mellanox-cx3.42618/#post-404650