Proxmox node works at 2.5GbE upstream but capped at 1GbE downstream

[samstringerhye]

When running iperf3 between devices on my LAN I am having an issue with my network throughput. I have a Beelink EQ12 with Intel i225v chipest and when connected to a 10GbE switch via SFP+ tranceiver I am able to get 2.5GbE throughput on the upstream to iperf3 servers but when going downstream it is showing it being capped at 1GbE. I have updated to the latest kernel and even tested a 2.5GbE USB-C adapter and still having the same issues.

iperf3 tests​

Accepted connection from 192.168.10.55, port 53132
[  5] local 192.168.10.1 port 5201 connected to 192.168.10.55 port 53142
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   266 MBytes  2.23 Gbits/sec                 
[  5]   1.00-2.00   sec   281 MBytes  2.35 Gbits/sec                 
[  5]   2.00-3.00   sec   281 MBytes  2.35 Gbits/sec                 
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec                 
[  5]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec                 
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec                 
[  5]   6.00-7.00   sec   281 MBytes  2.35 Gbits/sec                 
[  5]   7.00-8.00   sec   281 MBytes  2.35 Gbits/sec                 
[  5]   8.00-9.00   sec   281 MBytes  2.35 Gbits/sec                 
[  5]   9.00-10.00  sec   279 MBytes  2.34 Gbits/sec                 
[  5]  10.00-10.04  sec  10.8 MBytes  2.35 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.04  sec  2.73 GBytes  2.34 Gbits/sec                  receiver

[  5] local 192.168.10.1 port 5201 connected to 192.168.10.55 port 41498
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  91.3 MBytes   766 Mbits/sec  1380   18.4 KBytes       
[  5]   1.00-2.00   sec  91.9 MBytes   771 Mbits/sec  1385   19.8 KBytes       
[  5]   2.00-3.00   sec  91.5 MBytes   767 Mbits/sec  1314   18.4 KBytes       
[  5]   3.00-4.00   sec  96.6 MBytes   810 Mbits/sec  1553   18.4 KBytes       
[  5]   4.00-5.00   sec  95.6 MBytes   802 Mbits/sec  1391   18.4 KBytes       
[  5]   5.00-6.00   sec  92.0 MBytes   771 Mbits/sec  1435   22.6 KBytes       
[  5]   6.00-7.00   sec  92.6 MBytes   776 Mbits/sec  1334   39.6 KBytes       
[  5]   7.00-8.00   sec  96.4 MBytes   808 Mbits/sec  1442   18.4 KBytes       
[  5]   8.00-9.00   sec  94.9 MBytes   796 Mbits/sec  1270   22.6 KBytes       
[  5]   9.00-10.00  sec  93.1 MBytes   781 Mbits/sec  1148   17.0 KBytes       
[  5]  10.00-10.04  sec  3.59 MBytes   758 Mbits/sec   59   17.0 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   939 MBytes   785 Mbits/sec  13711             sender

 

[aaron]

Overall I would recommend to use iperf instead of iperf3. The naming is a bit confusing, but iperf3 will always be single threaded, while iperf will use multiple threads if paralell -P connections are configured.

Have you verified, for example with ethtool that all NICs are using 2.5 Gbit? If a cable might be bad, 1 Gbit is the fasted both side can negotiate. Also, does the 10 Gbit support 2.5 Gbit? AFAIU, only select newer switches support 1, 2.5, 5 and 10 Gbit. Older ones will usually support 100M, 1G and 10G, but nothing in between.