NetXtreme II BCM57810 10 Gigabit Ethernet slow

C

Carlos Chavez

Active Member
Proxmox Subscriber
Mar 2, 2018
14
0
41
54
I have a Dell R730xd with a NetXtreme II BCM57810 10 Gigabit Ethernet card. This connects my proxmox server to a Unifi Switch at 10gb using a DAC cable. Since I do not have any other 10g capable machines in the lab I can only test 2.5g speeds but even then I am only getting around 300mb/s on outgoing connections.

Here are some iperf3 tests:

(base) cursor@Cursor-MBP-19 ~ % iperf3 -c pve1
Connecting to host pve1, port 5201
[ 6] local 192.168.2.248 port 49936 connected to 192.168.2.70 port 5201
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-1.00 sec 211 MBytes 1.77 Gbits/sec
[ 6] 1.00-2.00 sec 281 MBytes 2.36 Gbits/sec
[ 6] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec
[ 6] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec
[ 6] 4.00-5.00 sec 280 MBytes 2.35 Gbits/sec
[ 6] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec
[ 6] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec
[ 6] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec
[ 6] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec
[ 6] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-10.00 sec 2.67 GBytes 2.29 Gbits/sec sender
[ 6] 0.00-10.00 sec 2.67 GBytes 2.29 Gbits/sec receiver

iperf Done.

(base) cursor@Cursor-MBP-19 ~ % iperf3 -R -c pve1
Connecting to host pve1, port 5201
Reverse mode, remote host pve1 is sending
[ 6] local 192.168.2.248 port 49942 connected to 192.168.2.70 port 5201
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-1.00 sec 33.3 MBytes 280 Mbits/sec
[ 6] 1.00-2.00 sec 50.0 MBytes 419 Mbits/sec
[ 6] 2.00-3.00 sec 34.1 MBytes 285 Mbits/sec
[ 6] 3.00-4.00 sec 44.9 MBytes 378 Mbits/sec
[ 6] 4.00-5.00 sec 36.1 MBytes 303 Mbits/sec
[ 6] 5.00-6.00 sec 57.0 MBytes 479 Mbits/sec
[ 6] 6.00-7.00 sec 42.8 MBytes 359 Mbits/sec
[ 6] 7.00-8.00 sec 48.4 MBytes 406 Mbits/sec
[ 6] 8.00-9.00 sec 36.5 MBytes 306 Mbits/sec
[ 6] 9.00-10.00 sec 41.8 MBytes 350 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 6] 0.00-10.00 sec 426 MBytes 357 Mbits/sec 4618 sender
[ 6] 0.00-10.00 sec 425 MBytes 357 Mbits/sec receiver

iperf Done.

83:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
83:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)

I have tried to change DAC cables and the port on the switch but still have the same problem. I tested with MTU at 1492, 1500 and 9000. Tested Jumbo frames on and off on the switch. Why would the outgoing speed be restricted? What more can I check? Running Proxmox 9.1 with 256GB of ram and 2 Xeon cpus. Any suggestions?
 
Hi,

have you tried using the -P switch to start multiple, parallel streams? Commonly this is due to the host being CPU-bound.

Additionally, you can check whether NIC hardware-offloading is available and/or enabled, that can also help when the CPU is the bottleneck.
 
cheiss said:
Hi,

have you tried using the -P switch to start multiple, parallel streams? Commonly this is due to the host being CPU-bound.

Additionally, you can check whether NIC hardware-offloading is available and/or enabled, that can also help when the CPU is the bottleneck.
Click to expand...
Thank you for the suggestion.

I used ethtool to turn off offloading because I saw many posts about the subject:
ethtool -K nic4 tso off rx-checksumming off tx-checksumming off lro off rx-vlan-offload off tx-vlan-offload off

Here is iperf3 with -P 2:

(base) cursor@Cursor-MBP-19 ~ % iperf3 -P 2 -R -c pve1
Connecting to host pve1, port 5201
Reverse mode, remote host pve1 is sending
[ 6] local 192.168.2.248 port 49430 connected to 192.168.2.70 port 5201
[ 8] local 192.168.2.248 port 49431 connected to 192.168.2.70 port 5201
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-1.00 sec 40.9 MBytes 343 Mbits/sec
[ 8] 0.00-1.00 sec 41.3 MBytes 346 Mbits/sec
[SUM] 0.00-1.00 sec 82.2 MBytes 689 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 1.00-2.00 sec 20.1 MBytes 169 Mbits/sec
[ 8] 1.00-2.00 sec 44.6 MBytes 374 Mbits/sec
[SUM] 1.00-2.00 sec 64.7 MBytes 542 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 2.00-3.00 sec 35.0 MBytes 294 Mbits/sec
[ 8] 2.00-3.00 sec 41.0 MBytes 344 Mbits/sec
[SUM] 2.00-3.00 sec 76.0 MBytes 638 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 3.00-4.00 sec 38.8 MBytes 326 Mbits/sec
[ 8] 3.00-4.00 sec 41.1 MBytes 345 Mbits/sec
[SUM] 3.00-4.00 sec 79.9 MBytes 670 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 4.00-5.00 sec 41.4 MBytes 347 Mbits/sec
[ 8] 4.00-5.00 sec 37.2 MBytes 312 Mbits/sec
[SUM] 4.00-5.00 sec 78.6 MBytes 659 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 5.00-6.01 sec 25.5 MBytes 213 Mbits/sec
[ 8] 5.00-6.01 sec 39.2 MBytes 327 Mbits/sec
[SUM] 5.00-6.01 sec 64.7 MBytes 540 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 6.01-7.00 sec 33.4 MBytes 282 Mbits/sec
[ 8] 6.01-7.00 sec 29.2 MBytes 246 Mbits/sec
[SUM] 6.01-7.00 sec 62.6 MBytes 528 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 7.00-8.00 sec 33.9 MBytes 284 Mbits/sec
[ 8] 7.00-8.00 sec 25.5 MBytes 214 Mbits/sec
[SUM] 7.00-8.00 sec 59.4 MBytes 498 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 8.00-9.00 sec 32.6 MBytes 273 Mbits/sec
[ 8] 8.00-9.00 sec 39.9 MBytes 335 Mbits/sec
[SUM] 8.00-9.00 sec 72.5 MBytes 608 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 9.00-10.00 sec 39.6 MBytes 332 Mbits/sec
[ 8] 9.00-10.00 sec 32.8 MBytes 275 Mbits/sec
[SUM] 9.00-10.00 sec 72.4 MBytes 607 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 6] 0.00-10.00 sec 342 MBytes 287 Mbits/sec 3197 sender
[ 6] 0.00-10.00 sec 341 MBytes 286 Mbits/sec receiver
[ 8] 0.00-10.00 sec 372 MBytes 312 Mbits/sec 3407 sender
[ 8] 0.00-10.00 sec 372 MBytes 312 Mbits/sec receiver
[SUM] 0.00-10.00 sec 714 MBytes 599 Mbits/sec 6604 sender
[SUM] 0.00-10.00 sec 713 MBytes 598 Mbits/sec receiver

Why would this only affect outgoing traffic and not incoming? Incoming gets the full 2.5g my other computer is capable of.

(base) cursor@Cursor-MBP-19 ~ % iperf3 -P 2 -c pve1
Connecting to host pve1, port 5201
[ 6] local 192.168.2.248 port 49440 connected to 192.168.2.70 port 5201
[ 8] local 192.168.2.248 port 49441 connected to 192.168.2.70 port 5201
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-1.00 sec 142 MBytes 1.19 Gbits/sec
[ 8] 0.00-1.00 sec 142 MBytes 1.19 Gbits/sec
[SUM] 0.00-1.00 sec 284 MBytes 2.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 1.00-2.00 sec 140 MBytes 1.18 Gbits/sec
[ 8] 1.00-2.00 sec 140 MBytes 1.18 Gbits/sec
[SUM] 1.00-2.00 sec 281 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 2.00-3.00 sec 140 MBytes 1.18 Gbits/sec
[ 8] 2.00-3.00 sec 140 MBytes 1.18 Gbits/sec
[SUM] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 3.00-4.00 sec 139 MBytes 1.17 Gbits/sec
[ 8] 3.00-4.00 sec 139 MBytes 1.17 Gbits/sec
[SUM] 3.00-4.00 sec 279 MBytes 2.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 4.00-5.00 sec 140 MBytes 1.18 Gbits/sec
[ 8] 4.00-5.00 sec 140 MBytes 1.18 Gbits/sec
[SUM] 4.00-5.00 sec 281 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 5.00-6.00 sec 140 MBytes 1.18 Gbits/sec
[ 8] 5.00-6.00 sec 140 MBytes 1.18 Gbits/sec
[SUM] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 6.00-7.00 sec 140 MBytes 1.18 Gbits/sec
[ 8] 6.00-7.00 sec 140 MBytes 1.17 Gbits/sec
[SUM] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 7.00-8.00 sec 140 MBytes 1.18 Gbits/sec
[ 8] 7.00-8.00 sec 140 MBytes 1.18 Gbits/sec
[SUM] 7.00-8.00 sec 281 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 8.00-9.00 sec 140 MBytes 1.18 Gbits/sec
[ 8] 8.00-9.00 sec 140 MBytes 1.18 Gbits/sec
[SUM] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 6] 9.00-10.00 sec 140 MBytes 1.18 Gbits/sec
[ 8] 9.00-10.00 sec 139 MBytes 1.17 Gbits/sec
[SUM] 9.00-10.00 sec 279 MBytes 2.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec sender
[ 6] 0.00-10.00 sec 1.37 GBytes 1.17 Gbits/sec receiver
[ 8] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec sender
[ 8] 0.00-10.00 sec 1.37 GBytes 1.17 Gbits/sec receiver
[SUM] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec sender
[SUM] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec receiver

iperf Done.
 
Last edited:
Carlos Chavez said:
I used ethtool to turn off offloading because I saw many posts about the subject:
Click to expand...
WIth hardware offloading turned off, networking can quickly become CPU-bound (or even memory-speed bound) for 1+Gbps, especially on bit older Xeons (you don't mention which, but based on the box itself).

Did you encounter problems with hardware offloading? Or is there any other particular reason why you turned it off?

Carlos Chavez said:
Why would this only affect outgoing traffic and not incoming?
Click to expand...
Sending simply involves a lot more steps, e.g. packets have to be buffered, put together, headers calculated etc.
 
cheiss said:
WIth hardware offloading turned off, networking can quickly become CPU-bound (or even memory-speed bound) for 1+Gbps, especially on bit older Xeons (you don't mention which, but based on the box itself).

Did you encounter problems with hardware offloading? Or is there any other particular reason why you turned it off?


Sending simply involves a lot more steps, e.g. packets have to be buffered, put together, headers calculated etc.
Click to expand...
I turned off hardware offloading after I started diagnosing the problem. Most of the posts dealing with similar issues recommend this. I have tried with offloading enabled and disabled but outgoing bandwidth is still the same at around only 300mbps while incoming is at the max I can measure of 2.5gbps
 
You must log in or register to reply here.
Top Bottom