Here is how i was able to get proxmox working with Infiniband and SR-IOV.
Hardware used is a mellanox switch (sx6036) and a mellanox Cx-4 100gbps EDR dual (or single) port card. Make sure the firmware is latest.
AS FAR AS IM AWARE, THIS WILL NOT WORK WITH OPENSM AND MUST HAVE A MELLANOX SWITCH FOR SR-IOV. on your switch, enable SM and virtualization, then restart the SM service by toggling it on or off, or rebooting the switch.
- ib sm enable
- ib sm virt enable
- configuration write
once that is configured you need to set up IOMMU and enable SRIOV on the hardware, this process will change depending on hardware so it will not be covered in this tutorial, the proxmox IOMMU config can be done by following this:
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
once that is completed you will want to install the following packages:
check for link, nodes and run diags:
Identifying the mellanox card bus and querying it:
enable SRIOV and 4 VF's (or however many you want).
Next steps are courtesy of Jose-d:
paste in the following, MAKE SURE TO UPDATE THE HCA:
Now enable the service:
Next we will create the script:
Paste in the following:
make sure you change permissions for the file:
now SR-IOV is configured and ready to attach to the VM.
From the VM we can now see full link on the SR-IOV device and it is in an active state:
Hardware used is a mellanox switch (sx6036) and a mellanox Cx-4 100gbps EDR dual (or single) port card. Make sure the firmware is latest.
AS FAR AS IM AWARE, THIS WILL NOT WORK WITH OPENSM AND MUST HAVE A MELLANOX SWITCH FOR SR-IOV. on your switch, enable SM and virtualization, then restart the SM service by toggling it on or off, or rebooting the switch.
- ib sm enable
- ib sm virt enable
- configuration write
once that is configured you need to set up IOMMU and enable SRIOV on the hardware, this process will change depending on hardware so it will not be covered in this tutorial, the proxmox IOMMU config can be done by following this:
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
once that is completed you will want to install the following packages:
apt install -y infiniband-diags ibutils rdma-core rdmacm-utils mstflintcheck for link, nodes and run diags:
- ibstat - MAKE NOTE OF THE HCA HERE
- ibnodes
- ibdiagnet
Identifying the mellanox card bus and querying it:
lspci | grep -i mellanoxmstflint -d <bus id here> qenable SRIOV and 4 VF's (or however many you want).
mstconfig -d <bus id here> set SRIOV_EN=1 NUM_OF_VFS=4Next steps are courtesy of Jose-d:
vim /etc/systemd/system/mellanox_initvf.service paste in the following, MAKE SURE TO UPDATE THE HCA:
Code:
[Unit]
After=network.target
[Service]
Type=oneshot
# note: change according to your hardware:
ExecStart=/bin/bash -c "/usr/bin/echo 4 > /sys/class/infiniband/<HCA HERE>/device/sriov_numvfs"
ExecStart=/usr/local/bin/initIbGuids.sh
StandardOutput=journal
TimeoutStartSec=60
RestartSec=60
[Install]
WantedBy=multi-user.targetNow enable the service:
systemctl enable mellanox_initvf.serviceNext we will create the script:
vim /usr/local/bin/initIbGuids.shPaste in the following:
Code:
#!/bin/bash
first_dev=$(ibstat --list_of_cas | head -n 1)
node_guid=$(ibstat ${first_dev} | grep "Node GUID" | cut -d ':' -f 2 | xargs | cut -d 'x' -f 2)
port_guid=$(ibstat ${first_dev} | grep "Port GUID" | cut -d ':' -f 2 | xargs | cut -d 'x' -f 2)
echo "first dev: $first_dev"
echo "node guid: $node_guid"
echo "port_guid: $port_guid"
if ip link show $first_dev &> /dev/null ; then
for vf in {0..3}; do
vf_guid=$(echo "${port_guid::-5}cafe$((vf+1))" | sed 's/..\B/&:/g')
echo "vf_guid for vf $vf is $vf_guid"
ip link set dev ${first_dev} vf $vf port_guid ${vf_guid}
ip link set dev ${first_dev} vf $vf node_guid ${vf_guid}
ip link set dev ${first_dev} vf $vf state auto
done
fimake sure you change permissions for the file:
chmod 777 /usr/local/bin/initIbGuids.shnow SR-IOV is configured and ready to attach to the VM.
From the VM we can now see full link on the SR-IOV device and it is in an active state:
Last edited:
