Proxmox MLNX_OFED and priority flow control issues

[myro1]

Hello,

I was hoping to get some assistance getting Mellanox drivers working correctly in proxmox. I am evaluating proxmox 7.4.3 (kernel PVE 5.15.107-1) for use in our environment but in order to be able to successfully use it I need to ensure priority flow control (pfc) is enabled in our network cards (connectX5 and connectX6 cards). This is needed for RoCE to perform in a lossless setup.

By default pfc is disabled and needs to be enabled using the mlnx_qos tool. You cannot configure pfc using the Nvidia Firmware Tools (MFT), as far as I'm aware mlnx_qos is not available there.

When trying to install the MLNX_OFED drivers to provide access to mlnx_qos (required to configure pfc) the install script prompts to remove a list of pve packages which would break pve.

What is the recommend/supported method of getting a PVE host running with functioning MLNX OFED?

Thanks for the help!

 

[cyfdecyf]

I encount the same problem. The solution I'm now using is download MLNX OFED package and install the deb package manually.

Specifically, on PVE 8 I'm using,

1. download MLNX OFED for Debian 11.3 (latest Debian version supported now), extract thet package
2. the individual deb packages are in the DEBS directory
3. install mlnx-tools package, e.g. dpkg -i mlnx-tools_23.07-0.2307050_amd64.deb.

For other components like kernel driver for Mellanox NIC, I'm just using kernel version or install with apt.

I found this Mellanox's mlnx-tools project, guess this is source of the deb package. We can see the mlnx_qos command is just a python script.