Mellanox ConnectX-6 (100GbE) Performance Issue – Only Reaching ~19Gbps Between Nodes

[jjuancsm3]

Hello everyone,


We’ve been encountering an issue during some pre-production testing, and I’d like to ask for your input — it seems likely to be a configuration-related problem, but we’re beginning to suspect there might be a hardware factor involved as well.


The setup involves connecting two nodes via recently acquired transceivers. According to our distributor, we should be able to achieve 100Gbps throughput between them. However, during our tests, the maximum bandwidth we’ve been able to reach is around 18–19Gbps.


We haven’t made any major changes to the configuration other than running some basic network throughput tests directly from the Proxmox shell. Unfortunately, those didn’t help resolve the issue.


I've attached a diagram showing how the two nodes are connected, along with some shell test outputs. Our main objective is to get as close as possible to the 100Gbps mark and rule out any physical or hardware limitations.


Network interface cards in use:

• Mellanox PN: MCX653106A-EACT-SP / ConnectX-6 EDR/HDR100/100GbE

If anyone has experienced similar behavior or has suggestions on where to look, we’d appreciate any guidance.


Thanks in advance.

 

[spirit]

how do you test ? if you use iperf3, be carefull that it's not multithreaded (even if you use -P option for multiple streams), until you use a really really recent version of iperf3, so you could be cpu/core limited. (just do a top to verify).

iperf2 is multithreaded if you want to compare

 

[ness1602]

Thats okay for single threaded iperf3, around 20-25gbp/s.