10G slow incoming

[sukerman]

I have a Qotom router with Intel(R) Atom(TM) CPU C3758R @ 2.40GHz.

It has 5 2.5G ports, and 4 10G SFP+ ports

04:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
06:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
07:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
08:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)

0b:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
0b:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
0c:00.0 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
0c:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)

The 2.5 ports work fine. The X553 ports did not work out of the box and I had to build and installed Intel ixgbe-5.20.9 drivers and it started working.

But while working, its only good outgoing.

root@qotom:~# ethtool eno4
Settings for eno4:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: yes

I have another host connected to the same switch, it has a 2.5G port.

root@qotom:~# iperf3 -c 10.1.10.1 -P 3
Connecting to host 10.1.10.1, port 5201
[  5] local 10.1.0.1 port 48210 connected to 10.1.10.1 port 5201
[  7] local 10.1.0.1 port 48212 connected to 10.1.10.1 port 5201
[  9] local 10.1.0.1 port 48222 connected to 10.1.10.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  72.0 MBytes   604 Mbits/sec  189    115 KBytes
[  7]   0.00-1.00   sec  89.4 MBytes   750 Mbits/sec  182    236 KBytes
[  9]   0.00-1.00   sec   106 MBytes   892 Mbits/sec  154    287 KBytes
[SUM]   0.00-1.00   sec   268 MBytes  2.25 Gbits/sec  525

This looks fine, but going the other way from that host (or indeed any host, including from the internet)....

root@glovary:~# iperf3 -c 10.1.0.1 -P 3
Connecting to host 10.1.0.1, port 5201
[  5] local 10.1.10.1 port 58228 connected to 10.1.0.1 port 5201
[  7] local 10.1.10.1 port 58244 connected to 10.1.0.1 port 5201
[  9] local 10.1.10.1 port 58252 connected to 10.1.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  91.9 KBytes   752 Kbits/sec    3   1.41 KBytes
[  7]   0.00-1.00   sec  91.9 KBytes   752 Kbits/sec    7   1.41 KBytes
[  9]   0.00-1.00   sec  91.9 KBytes   752 Kbits/sec   10   2.83 KBytes
[SUM]   0.00-1.00   sec   276 KBytes  2.26 Mbits/sec   20
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    7   1.41 KBytes
[  7]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec   10   1.41 KBytes
[  9]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec   10   2.83 KBytes
[SUM]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec   27
- - - - - - - - - - - - - - - - - - - - - - - - -

I get the same if I try to scp, or speedtest.

Setting the interface to 1G makes no difference

root@qotom:~# ethtool -s eno4 autoneg off speed 1000 duplex full
root@qotom:~# speedtest
Retrieving speedtest.net configuration...
Testing from Avatel Telecom (45.xx.xx.xx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Scan Sat Network SL (Torrevieja) [18.31 km]: 19.165 ms
Testing download speed................................................................................
Download: 0.12 Mbit/s
Testing upload speed......................................................................................................
Upload: 9.58 Mbit/s

The interface shows a load of errors:

root@qotom:~# ethtool -S eno4
NIC statistics:
     rx_packets: 1309849
     tx_packets: 2931477
     rx_bytes: 745125695
     tx_bytes: 4421460228
     rx_errors: 472918
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     multicast: 1954
     collisions: 0
     rx_over_errors: 0
     rx_crc_errors: 453204
     rx_frame_errors: 0
     rx_fifo_errors: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     rx_pkts_nic: 1309864
     tx_pkts_nic: 2931477
     rx_bytes_nic: 750366671
     tx_bytes_nic: 4433189430
     lsc_int: 2
     tx_busy: 0
     non_eop_descs: 0
     broadcast: 4321
     rx_no_buffer_count: 0
     tx_timeout_count: 0
     tx_restart_queue: 0
     rx_length_errors: 374

Things also just generally lock up too. If I try service pve-firewall stop the machine locks and thats it.

Any ideas welcome, I've tried Hi Fiber SFP adaptors and QSFPTEK which are supposed to be Intel compatible, but they both behave the same.

Thanks,

 

[guggero]

Is this the Qotom Q20331G9 (or similar) by any chance?

I have the exact same problem. I tried two different 10GBit/s Ethernet modules.
TX gives me the full speed (~9.9 GBit/s) but RX is super slow (~20 MBit/s).

And I see a very high number for rx_crc_errors as well.

Would love to be able to debug this further, but don't have any idea where to start.
Can't even tell if this might be a hardware issue or a kernel driver problem.

What exact kernel version are you running?
I see this on Linux OpenWrt 5.15.150 (ixgbe 5.15.150).

 

[sukerman]

I think the problem is that the ethernet modules at least the ones I've tried draw too much power. Try it with some DAC interfaces, I seem to get 10G both ways now if I use the latest proxmox kernel.

 

[guggero]

I think the problem is that the ethernet modules at least the ones I've tried draw too much power. Try it with some DAC interfaces, I seem to get 10G both ways now if I use the latest proxmox kernel.

Thank you very much for your response! It helped me to resolve the issue by switching from Ethernet modules to DAC interfaces.
Everything works well and on full speed now!