Proxmox MLNX_OFED and priority flow control issues

myro1

New Member
Jun 26, 2023
2
0
1
Hello,

I was hoping to get some assistance getting Mellanox drivers working correctly in proxmox. I am evaluating proxmox 7.4.3 (kernel PVE 5.15.107-1) for use in our environment but in order to be able to successfully use it I need to ensure priority flow control (pfc) is enabled in our network cards (connectX5 and connectX6 cards). This is needed for RoCE to perform in a lossless setup.

By default pfc is disabled and needs to be enabled using the mlnx_qos tool. You cannot configure pfc using the Nvidia Firmware Tools (MFT), as far as I'm aware mlnx_qos is not available there.

When trying to install the MLNX_OFED drivers to provide access to mlnx_qos (required to configure pfc) the install script prompts to remove a list of pve packages which would break pve.

What is the recommend/supported method of getting a PVE host running with functioning MLNX OFED?

Thanks for the help!
 
Last edited:
I encount the same problem. The solution I'm now using is download MLNX OFED package and install the deb package manually.

Specifically, on PVE 8 I'm using,

  1. download MLNX OFED for Debian 11.3 (latest Debian version supported now), extract thet package
  2. the individual deb packages are in the DEBS directory
  3. install mlnx-tools package, e.g. dpkg -i mlnx-tools_23.07-0.2307050_amd64.deb.

For other components like kernel driver for Mellanox NIC, I'm just using kernel version or install with apt.

I found this Mellanox's mlnx-tools project, guess this is source of the deb package. We can see the mlnx_qos command is just a python script.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!