Mellanox Drivers for PVE 5.4

Discussion in 'Proxmox VE: Networking and Firewall' started by gdi2k, Apr 27, 2019.

  1. gdi2k

    gdi2k Member

    Aug 13, 2016
    Likes Received:
    We're using Proxmox Virtual Environment version 5.4-3 and have upgraded our cluster to Mellanox Connectx-3 40G NICs.

    We would like to use the Mellanox drivers if possible, but their download page only provides drivers for Debian up to version 9.5 and I believe PVE 5.4-3 runs on top of Debian 9.8.

    How are others getting around this issue?

    We only seem to be able to achieve 20 GBPS (iperf3) using the included open source drivers. The NICs are connected to Arista 7050 series 40G switches so I was hoping for wire speed. I don't know if this is a driver related issue or something else, but drivers seemed like a good place to start.
  2. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Jul 28, 2015
    Likes Received:
    Actually, this is a DKMS driver, so you need to match our kernel, which for 5.x is now the one from Ubuntu 18.04 LTS, which they list as supported.
    But then, the Debian 9.5 and the current Ubuntu 18.04 downloads get you the exact same source tar archive (I compared md5sums).

    So, you either download the prebuild tar archive with DEBS included, or the source package and build them yourself. The latter naturally isn't conforming to Debian/Ubuntu build systems so yeah, but a `fakeroot ./install` should get you some debian packages (and hints about build-dependencies), fakeroot as I highly suggest not running such custom stuff as real root.
    Additionally, I'd recommend building this in a small Debian/Ubuntu VM - so the host PVE does not get touched, and then installed

    If you use the prebuild (IMO worth a try), then just install them on a PVE host (ideally one containing the NICs but not being in production/required at the moment).
    As this are DKMS packages, they should handle the local kernel, and rerun on every ABI bump of our kernel.
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. gdi2k

    gdi2k Member

    Aug 13, 2016
    Likes Received:
    Many thanks for this - it worked as you suggested using the included debs (DEBS_ETH directory for ethernet only in my case). Chapter 2.8.2 of the manual can be followed to install (add the DEBS_ETH directory as an apt-repo, add their key, update and install).

    dkms status
    mlnx-en, 4.5: added

    After a reboot everything seems normal but based on the output of ethtool, it doesn't seem like the new drivers are actually being used:

    driver: mlx4_en
    version: 4.0-0
    firmware-version: 2.42.5000
    bus-info: 0000:0b:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: no
    supports-register-dump: no
    supports-priv-flags: yes

    This is the same output as I get for the servers without the drivers installed. I would have expected it to say version 4.5.

    Are there additional steps required to switch from the included open source driver to the newly installed Mellanox driver?
  4. robertb

    robertb New Member

    Apr 4, 2017
    Likes Received:
    update your initramfs:
    update-initramfs -k all -u -v
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice