Hey folks,
Got real darn tired of trying to poke and prod and guess and then forget a month later what i did in the first place on a new server so im including my guide to get your mellanox connectx-3 card up and running on proxmox 7.3
Ultimately it looks like the real problem/struggle I dealt with that wasn't part of the official nvidia guides was two things, 1) that im using the OS version of the drivers instead of OFED which obviously on proxmox 7.3 and debian 11 is not supported and never will be now. Then 2) That the kernel is only loading the mlx4_core module and neither mlx4_en or mlx4_ib for ethernet/infiniband respectively after it.
Im using an HP dl360p G8, wanting to run the dual-port MCX312A-XCBT card in ethernet mode. with 4 probe able VFS on either port you'll have to navigate the hell that is nvidias migration of mellanox's site to get the backup/recovery stuff for your version if it doesn't match.
NOTE: I didn't know this and felt real stupid for awhile. once you have your VF's they are assignable in alternating order so .0 is port 1 .1 is port 2 .2 is port 1 .3 is port 2, etc...
I haven't worked with linux since pre-ubuntu other than very casually and will not be going deep into it anyways so i just need things to work and be simplified hence this guide in case i loose my notes or save someone else from the timekiller it is trying to figure out how with the same lack of experience and trying to just get a homelab started.
####start of guide####
--Do all your normal bios and base proxmox configs to turn on sriov/pass thru support--
--download newer firmware for recovery and tools will probably only use MFT and mlxup yourself but i gotta tired of navigating the hellscape of mellanox/nvidia finding em so im including them--
wget http://www.mellanox.com/downloads/f...00-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin.zip
wget https://www.mellanox.com/downloads/firmware/mlxup/4.22.1/SFX/linux_x64/mlxup
wget https://www.mellanox.com/downloads/MFT/mft-4.22.1-11-x86_64-deb.tgz
wget https://github.com/Mellanox/mstflint/releases/download/v4.22.0-1/mstflint-4.22.0-1.tar.gz
--Other way can try to grab the mft tools from deb itself, I downloaded and built myself so didn't test it--
sudo apt install mstflint
--otherwise use the tar ball you grabbed--
apt-get install gcc make dkms pve-headers
tar -xvzf mft-4.22.1-11-x86_64-deb.tgz
cd mft-4.22.1-11-x86_64-deb/
./install.sh
mst start
--verify card there--
mst status
--update firmware--
chmod +x mlxup
./mlxup
--Check existing fw config and add proper sriov support--
mlxconfig -d /dev/mst/mt4099_pciconf0 q
mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
--now reboot, Card FW ready for SRIOV and vfs time to do the driver--
--Add mlx4_core.conf to enable SRIOV and VFS on driver--
--The first line is the only one you really need, since im running ethernet I added the others for performance tuning per mellanox docs--
pico /etc/modprobe.d/mlx4_core.conf
--add these to the file, again line one is the only mandatory one--
options mlx4_core num_vfs=4,4,0 port_type_array=2,2 probe_vf=4,4,0 probe_vf=4,4,0
options mlx4_core enable_sys_tune=1
options mlx4_en inline_thold=0
options mlx4_core log_num_mgm_entry_size=-7
--OPTION 1: I havent tested i was so annoyed that once I got it working I wasn't in the mood to undo and redo--
--All you may have to do next is--
update-initramfs -u
--OPTION 2: If that doesn't work then go with the following--
modprobe -r mlx4_en mlx4_ib
modprobe mlx4_en
update-initramfs -u
--verify all the vfs show up--
ip link
--reboot and confirm vfs still there at startup
lspci -vvv | grep Mellanox
--All done--
--if needed for later to attempt recovery use there github mstflint tool--
tar -xvzf mstflint-4.22.0-1.tar.gz
cd mstflint-4.22.0/
apt install zlib1g-dev openssl libssl-dev automake
--unfortunately I don't know what package to grab the infiniband lib from so this is purely ethernet and I exclude infiniband in the build
./configure --disable-inband
make
make install
####End of Guide####
Hope this helps
Got real darn tired of trying to poke and prod and guess and then forget a month later what i did in the first place on a new server so im including my guide to get your mellanox connectx-3 card up and running on proxmox 7.3
Ultimately it looks like the real problem/struggle I dealt with that wasn't part of the official nvidia guides was two things, 1) that im using the OS version of the drivers instead of OFED which obviously on proxmox 7.3 and debian 11 is not supported and never will be now. Then 2) That the kernel is only loading the mlx4_core module and neither mlx4_en or mlx4_ib for ethernet/infiniband respectively after it.
Im using an HP dl360p G8, wanting to run the dual-port MCX312A-XCBT card in ethernet mode. with 4 probe able VFS on either port you'll have to navigate the hell that is nvidias migration of mellanox's site to get the backup/recovery stuff for your version if it doesn't match.
NOTE: I didn't know this and felt real stupid for awhile. once you have your VF's they are assignable in alternating order so .0 is port 1 .1 is port 2 .2 is port 1 .3 is port 2, etc...
I haven't worked with linux since pre-ubuntu other than very casually and will not be going deep into it anyways so i just need things to work and be simplified hence this guide in case i loose my notes or save someone else from the timekiller it is trying to figure out how with the same lack of experience and trying to just get a homelab started.
####start of guide####
--Do all your normal bios and base proxmox configs to turn on sriov/pass thru support--
--download newer firmware for recovery and tools will probably only use MFT and mlxup yourself but i gotta tired of navigating the hellscape of mellanox/nvidia finding em so im including them--
wget http://www.mellanox.com/downloads/f...00-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin.zip
wget https://www.mellanox.com/downloads/firmware/mlxup/4.22.1/SFX/linux_x64/mlxup
wget https://www.mellanox.com/downloads/MFT/mft-4.22.1-11-x86_64-deb.tgz
wget https://github.com/Mellanox/mstflint/releases/download/v4.22.0-1/mstflint-4.22.0-1.tar.gz
--Other way can try to grab the mft tools from deb itself, I downloaded and built myself so didn't test it--
sudo apt install mstflint
--otherwise use the tar ball you grabbed--
apt-get install gcc make dkms pve-headers
tar -xvzf mft-4.22.1-11-x86_64-deb.tgz
cd mft-4.22.1-11-x86_64-deb/
./install.sh
mst start
--verify card there--
mst status
--update firmware--
chmod +x mlxup
./mlxup
--Check existing fw config and add proper sriov support--
mlxconfig -d /dev/mst/mt4099_pciconf0 q
mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
--now reboot, Card FW ready for SRIOV and vfs time to do the driver--
--Add mlx4_core.conf to enable SRIOV and VFS on driver--
--The first line is the only one you really need, since im running ethernet I added the others for performance tuning per mellanox docs--
pico /etc/modprobe.d/mlx4_core.conf
--add these to the file, again line one is the only mandatory one--
options mlx4_core num_vfs=4,4,0 port_type_array=2,2 probe_vf=4,4,0 probe_vf=4,4,0
options mlx4_core enable_sys_tune=1
options mlx4_en inline_thold=0
options mlx4_core log_num_mgm_entry_size=-7
--OPTION 1: I havent tested i was so annoyed that once I got it working I wasn't in the mood to undo and redo--
--All you may have to do next is--
update-initramfs -u
--OPTION 2: If that doesn't work then go with the following--
modprobe -r mlx4_en mlx4_ib
modprobe mlx4_en
update-initramfs -u
--verify all the vfs show up--
ip link
--reboot and confirm vfs still there at startup
lspci -vvv | grep Mellanox
--All done--
--if needed for later to attempt recovery use there github mstflint tool--
tar -xvzf mstflint-4.22.0-1.tar.gz
cd mstflint-4.22.0/
apt install zlib1g-dev openssl libssl-dev automake
--unfortunately I don't know what package to grab the infiniband lib from so this is purely ethernet and I exclude infiniband in the build
./configure --disable-inband
make
make install
####End of Guide####
Hope this helps
Last edited:
