Mellanox ConnectX-3 Pro (MT27520) performance

[uriunger]

I have upgraded the NIC on my main proxmox node (supermicro X10DRi-T4+ with 2xE5-2699 and 512GB memory) to Mellanox ConnectX-3. My windows desktop (i9 9920X with 64GB memory) has the same NIC. Both NICs are connected to 40gbe ports on a Brocade ICS6610 switch.

Running iperf3 tests between the two machines, I can only get to ~11Gbit. Concurrently running multiple iperf3 sessions does not show any higher throughout.

I am using default drivers on proxmox. I have tried to upgrade to Mellanox own drivers, but I am not sure how to do it correctly as Mellanox's bundle does not support proxmox.

Any recommendation on how to get higher throughput?

 

[Dunuin]

Maybe SR-IOV and then PCI passthrough of virtual functions into the VMs? You can also try to use jumboframes so there are bigger but less packets.
How fast virtio NICs and Linux bridges can operate depends on the single-threaded performance of your CPU. Modern CPUs usually can handle around 20Gbit.

 

[uriunger]

Thank you Dunuin.

I should have mentioned that I am testing against the Proxmox node itself - no VMs in this story. I should be getting the native NIC Linux performance.

 

[Neobin]

What is the CPU utilization on both systems while testing?
Do you see any core/thread packed at (nearly) 100% on any of the systems?

As Dunuin said, try again with jumbo frames.

 

[uriunger]

A single iperf session does reach high CPU utilization (~85-90%) on the server side. The windows client has a much faster single-threaded performance, so no high CPU was observed. When running a pair of iperf sessions, each iperf process on the server consumes ~40%; total bandwidth remains around 11-12Gbit, so it would seem the bottleneck is not CPU related.

Jumbo frames were enabled on NICs and the switch (9600 bytes).

 

[spirit]

can you try with iperf2 on both client/server side with -P option ? (iperf2 is multithreaded with parallel stream)

 

[uriunger]

As expected, iperf2 with -P (2/4/8 threads) gave the same results.