Updating firmware for Mellanox MT27520 Family [ConnectX-3 Pro] (aka cx322a) in Proxmox

X

xelar

New Member
Nov 26, 2023
19
0
1
I have 3 nearly identical nodes each with a Mellanox CX322A Dual SFP+ NIC. On my newest node I was easily able to pass through the NIC to Windows and I updated the firmware using an installer from DELL (the NICs are DELL rebranded). On my other two nodes (Lenovo Tiny PC m920q) I am unable to pass through the NIC as I keep getting:

Error: started failed: QEMU exited with code 1

...no matter what I tried to do. The only thing that I found that may potentially be causing an issue is that the NIC shared the same IOMMU group number as the PCIe controller.

Therefore, I downloaded a BIN of the firmware meant to be used on Linux by just executing in the SHELL to update the firmware, but that too does not work either. The error is:

/opt/dell/updatepackage/firmware.BIN-16002.TnsBM4/spsetup.sh: 1293: Syntax error: redirection unexpected

So... what are my options to update the firmware other than popping the cards into another computer?

The firmware I am using is here:
https://www.dell.com/support/home/en-pg/drivers/driversdetails?driverid=h9rjp
 
As shared by another user here , I needed to blacklist the modules that were interfering with passing through the PCIe NIC. Looking at the output of dmesg I only saw this error: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:01:00.0)) vs. 00000080 (i801_smbus)

The post linked above blacklisted the NIC itself and both i801 and smbus so that is what I did:

Code: 
nano /etc/modprobe.d/pve-blacklist.conf

# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb

#Disable following modules to be able to pass through the PCIe Mellanox NIC, then revert.
#Execute this command, and reboot: update-initramfs -u -k all
blacklist mlx4_core
blacklist i2c_i801
blacklist i2c_smbus

After adding the last 3 blacklisted modules, I ran this >> update-initramfs -u -k all to make it effective and rebooted.

I was finally able to start the Windows VM and the Mellanox NIC was successfully passed through to Windows! I therefore updated the NIC's firmware, reverted my changes in pve-blacklist.conf commenting out the modules I blacklisted, ran the update command, and rebooted one final time.

Sharing because I lost hours on this and this post may help others avoid the same. There is a lot of talk about this, but most of it left out details or gave many variations that made it more confusing.

Edit: This is how I did the pass through
1709924764178.png
 
Last edited:
To clarify, you updated the firmware with the Windows firmware? Do you know if using the Windows firmware will make it fail a pass through to Linux or FreeBSD? The reason I ask is that I'm trying to use it for OPNSense in a cheap Lenovo P330 (almost exactly your setup, just using a different OS).

If I passed it through to Ubuntu, then updated the firmware with the Linux firmware, maybe it would be fine?
 
You must log in or register to reply here.
Top Bottom