[SOLVED] 10G Ethernet with Intel X550T only working at 1Gbit/s

[Mike_TBF]

Hi everyone,

right now I am trying to get Proxmox running on some new Supermicro X10SRM-TF mainboards. They have dual 10G ethernet on board, connected to a Zyxel 10G switch; lspci reports the chipset as
Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
Networking works fine, but only at 1Gbit/s, not at 10.

ethtool reports the following:

        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: umbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Trying to force the speed to 10G does not work either:

~# ethtool -s eth0 speed 10000
Cannot advertise speed 10000
Cannot set new settings: Invalid argument
  not setting speed

Cabling is ok of course: brand new Cat6a cables that work well with an Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) card in a different machine connected to the same switch.

Right now I hope I simply missed a BIOS setting or such. Other than that: could the ixgbe module be too old for that chipset? It reports as version 4.4.6, whereas the current version downloadable from Intel seems to be 5.0.4. I already tried to compile it for testing purposes, but it failed:

:~/driver/ixgbe-5.0.4/src# make
make[1]: Entering directory '/usr/src/linux-headers-4.4.44-1-pve'
  CC [M]  /root/driver/ixgbe-5.0.4/src/ixgbe_main.o
In file included from /root/driver/ixgbe-5.0.4/src/ixgbe_osdep.h:38:0,
                 from /root/driver/ixgbe-5.0.4/src/ixgbe_type.h:66,
                 from /root/driver/ixgbe-5.0.4/src/ixgbe_dcb.h:28,
                 from /root/driver/ixgbe-5.0.4/src/ixgbe.h:45,
                 from /root/driver/ixgbe-5.0.4/src/ixgbe_main.c:52:
/root/driver/ixgbe-5.0.4/src/kcompat.h:5229:20: error: static declaration of ‘napi_consume_skb’ follows non-static declaration
 static inline void napi_consume_skb(struct sk_buff *skb,
                    ^
In file included from include/linux/if_ether.h:23:0,
                 from include/uapi/linux/ethtool.h:17,
                 from include/linux/ethtool.h:17,
                 from include/linux/netdevice.h:42,
                 from /root/driver/ixgbe-5.0.4/src/ixgbe_main.c:31:
include/linux/skbuff.h:2346:6: note: previous declaration of ‘napi_consume_skb’ was here
 void napi_consume_skb(struct sk_buff *skb, int budget);
      ^
scripts/Makefile.build:258: recipe for target '/root/driver/ixgbe-5.0.4/src/ixgbe_main.o' failed
make[2]: *** [/root/driver/ixgbe-5.0.4/src/ixgbe_main.o] Error 1
Makefile:1420: recipe for target '_module_/root/driver/ixgbe-5.0.4/src' failed
make[1]: *** [_module_/root/driver/ixgbe-5.0.4/src] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-4.4.44-1-pve'
Makefile:107: recipe for target 'default' failed
make: *** [default] Error 2

Is there anyone here who might point me in the right direction?

CYa
Mike

 

[mir]

You have tried to make a direct connect between two nics?

 

[Mike_TBF]

You have tried to make a direct connect between two nics?

Right now, they are connected to a 10G switch. I did not try a direct connection yet. I have a remote connection to the cluster, but no physical access during the weekend. So I can only test that on Monday.

 

[Mike_TBF]

sorry to bother you, mates. I found the cause of the issue.
The network adapters are connected to a Zyxel XS1920 switch, and that has a known interoperability issue with Intel X550 based network adapters. It is not enough to upgrade the switch's regular firmware though, but you must install the separate "PHY firmware" for its Broadcom ports.

Since this is a touchy issue, I'll try that later, but my exact problem (intel X550 peers only working at 1Gbit/s) is mentioned in the switch's firmware change log.

 

[mir]

That would have been my next suggestion;-) Switches, and in particular if using sfp, can be very piggy and refuse to enable 10 G.