[SOLVED] Infiniband Mellanox 40Gb Speed Low

Younex

Well-Known Member
Jan 30, 2018
92
21
48
43
Hi,

first, merry christmas guys!

I just installed 2 mellanox infiniband cards (40Gb Connectx-2) each on on server and installed after that the latest mellanox drivers from here:
http://www.mellanox.com/page/products_dyn?product_family=27

Finally i made a test with iperf but i get only this speed:
[ 4] 0.0-10.0 sec 5.17 GBytes 4.44 Gbits/sec

This is the card i use:
http://www.mellanox.com/page/products_dyn?product_family=4&mtag=connect_x
I see it only allows 5Gb/s?

This is lspci output:
24:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0)

So 5GT/s means for Ethernet mode?
I really confused what i can expect from the card and if this is already the maximum/limit of the card.

Do i need to use the built in modules instead of the drivers or are some parameters missing to get more speed or do i need an higher card ConnectX-3?

Thanks for help!
 
Last edited:
The 5GT/s refer to the PCI-E 2.0 speed (which is expressed in Gigatransfers/s) - see https://en.wikipedia.org/wiki/PCI_Express

AFAIR, you need PCI-E 2.0 x4 for 10Gb throughput.
I guess you'll get better iperf results, if you tweak the command-line parameters some more
(check your results in UDP mode, with an appropriate bandwidth command-line switch)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!