Mellanox Drivers for PVE 5.4

[gdi2k]

We're using Proxmox Virtual Environment version 5.4-3 and have upgraded our cluster to Mellanox Connectx-3 40G NICs.

We would like to use the Mellanox drivers if possible, but their download page only provides drivers for Debian up to version 9.5 and I believe PVE 5.4-3 runs on top of Debian 9.8.

How are others getting around this issue?

We only seem to be able to achieve 20 GBPS (iperf3) using the included open source drivers. The NICs are connected to Arista 7050 series 40G switches so I was hoping for wire speed. I don't know if this is a driver related issue or something else, but drivers seemed like a good place to start.

 

[t.lamprecht]

We would like to use the Mellanox drivers if possible, but their download page only provides drivers for Debian up to version 9.5 and I believe PVE 5.4-3 runs on top of Debian 9.8.

Actually, this is a DKMS driver, so you need to match our kernel, which for 5.x is now the one from Ubuntu 18.04 LTS, which they list as supported.
But then, the Debian 9.5 and the current Ubuntu 18.04 downloads get you the exact same source tar archive (I compared md5sums).

So, you either download the prebuild tar archive with DEBS included, or the source package and build them yourself. The latter naturally isn't conforming to Debian/Ubuntu build systems so yeah, but a `fakeroot ./install` should get you some debian packages (and hints about build-dependencies), fakeroot as I highly suggest not running such custom stuff as real root.
Additionally, I'd recommend building this in a small Debian/Ubuntu VM - so the host PVE does not get touched, and then installed

If you use the prebuild (IMO worth a try), then just install them on a PVE host (ideally one containing the NICs but not being in production/required at the moment).
As this are DKMS packages, they should handle the local kernel, and rerun on every ABI bump of our kernel.

 

[gdi2k]

Many thanks for this - it worked as you suggested using the included debs (DEBS_ETH directory for ethernet only in my case). Chapter 2.8.2 of the manual can be followed to install (add the DEBS_ETH directory as an apt-repo, add their key, update and install).

dkms status
mlnx-en, 4.5: added

After a reboot everything seems normal but based on the output of ethtool, it doesn't seem like the new drivers are actually being used:

driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5000
expansion-rom-version:
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

This is the same output as I get for the servers without the drivers installed. I would have expected it to say version 4.5.

Are there additional steps required to switch from the included open source driver to the newly installed Mellanox driver?

 

[robertb]

update your initramfs:
update-initramfs -k all -u -v