I am trying to setup a Mellanox ConnectX-3 infiniband with IBoP and hitting a speed issue. Same adapter works fine on supermicro servers without any issues. With 2 supermicro servers using iperf3 with multiple threads it goes over 30mbps in both directions without any issues. When using the Dell R630 as iperf3 client it hits 16gbps no more, it the dell side is setup as iperf3 server then the max speed I see is around 2.5gbps (it looks like it has something to do with the PCIe lanes being stuck at pcie 1.0 and only using 1 lane). I tried different settins with the BIOS and still no lock. (btw, MTU is set at max 65520). Did anyone installed one of these infiniband cards on a dell server and had success?
This can also be replicated with Connectx-2 cards with max speed capped at 26gbps between supermicro servers and 2.5gbps between dell servers. From Dell to supermicro reproduces the same results with connectx-3 cards. All the cards are connected to a mellanox IS5030 switch and all links come up as 40gbps links. Any ideas appreciated. Thanks.
This can also be replicated with Connectx-2 cards with max speed capped at 26gbps between supermicro servers and 2.5gbps between dell servers. From Dell to supermicro reproduces the same results with connectx-3 cards. All the cards are connected to a mellanox IS5030 switch and all links come up as 40gbps links. Any ideas appreciated. Thanks.
Last edited: