Mellanox 100gbe Link only getting ~55 gbit/s

Feb 24, 2022

I have 2 servers with each one Mellanox ConnectX-4 100 GbE QSFP installed. >I connected the 2 with a Mellanox 100gbe DAC cable. and both idracs show me a link with of 100gbit/s. However when I run ipref3 on both ends I only get around 34gbit/s. I have read that iperf3 might be limited to 1 cpu core only so I ran 3 parallel streams which results in the 3 streams getting 18-19gbit/s each so ~ 55gbit/s. which is better than 34gbit/s but far from 100gbits. what can I still try? do I have to enable any specific features in the drivers?

one of the servers is a
dell r730xd with dual Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz and 128gb ecc ram
and the other a
dell r740xd with dual Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz and 512gb ecc ram

the dell r730xd is running truenas Scale latest version
the dell r740xd is running proxmox VE latest version

are there any driver settings or something I need to change? like HW offlodaing or similar?

on the r740xd with dual Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz I tried doing the grub config with

intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll intel_pstate=disable

like stated here:

but I can't seem to get it working. Even with the line added top the grub config, after reboot it still shows "9" instead of "0"
when running
cat /sys/module/intel_idle/parameters/max_cstate

any help is appreciated


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!