[SOLVED] Infiniband Mellanox 40Gb Speed Low

[Younex]

Hi,

first, merry christmas guys!

I just installed 2 mellanox infiniband cards (40Gb Connectx-2) each on on server and installed after that the latest mellanox drivers from here:
http://www.mellanox.com/page/products_dyn?product_family=27

Finally i made a test with iperf but i get only this speed:
[ 4] 0.0-10.0 sec 5.17 GBytes 4.44 Gbits/sec

This is the card i use:
http://www.mellanox.com/page/products_dyn?product_family=4&mtag=connect_x
I see it only allows 5Gb/s?

This is lspci output:
24:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0)

So 5GT/s means for Ethernet mode?
I really confused what i can expect from the card and if this is already the maximum/limit of the card.

Do i need to use the built in modules instead of the drivers or are some parameters missing to get more speed or do i need an higher card ConnectX-3?

Thanks for help!

 

[Stoiko Ivanov]

The 5GT/s refer to the PCI-E 2.0 speed (which is expressed in Gigatransfers/s) - see https://en.wikipedia.org/wiki/PCI_Express

AFAIR, you need PCI-E 2.0 x4 for 10Gb throughput.
I guess you'll get better iperf results, if you tweak the command-line parameters some more
(check your results in UDP mode, with an appropriate bandwidth command-line switch)

 

[Younex]

Thank you Stoiko