[SOLVED] 10GB NIC only works in 1GB mode

[oz1cw7yymn]

Hi,

I am installing a Dell R720x with an X540 NIC. This card has two 10GB and 2 1GB, however they are all just working at 1GB. When I connect the card to a 10GB switch it negotiates 1GB only. I can't see any relevant issues in dmesg and I don't know if there is a way to disable/enable 10GB capability somehow. The kernel is using the bnx2x driver.

Here are the relevant outputs:

Spoiler: dmesg | grep bnx2

[ +0.216486] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.713.36-0 (2014/02/10)
[ +0.000278] bnx2x 0000:01:00.0: msix capability found
[ +0.000559] bnx2x 0000:01:00.0: part number 394D4342-30383735-30305430-473030
[ +0.094200] bnx2x 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ +0.000125] bnx2x 0000:01:00.1: msix capability found
[ +0.000371] bnx2x 0000:01:00.1: part number 394D4342-30383735-30305430-473030
[ +0.010540] bnx2x 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ +0.000083] bnx2x 0000:01:00.2: msix capability found
[ +0.000366] bnx2x 0000:01:00.2: part number 394D4342-30383735-30305430-473030
[ +0.057505] bnx2x 0000:01:00.2: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ +0.000120] bnx2x 0000:01:00.3: msix capability found
[ +0.000377] bnx2x 0000:01:00.3: part number 394D4342-30383735-30305430-473030
[ +0.005917] bnx2x 0000:01:00.3: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ +0.003137] bnx2x 0000:01:00.0 eno1: renamed from eth0
[ +0.020478] bnx2x 0000:01:00.1 eno2: renamed from eth2
[ +0.064607] bnx2x 0000:01:00.2 eno3: renamed from eth4
[ +0.014504] bnx2x 0000:01:00.3 eno4: renamed from eth6

Spoiler: lspci -s 01:00.1 -vvnnn

01:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57800 1/10 Gigabit Ethernet [14e4:168a] (rev 10)
Subsystem: Dell BCM57800 10-Gigabit Ethernet [1028:1f5d]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 64
NUMA node: 0
Region 0: Memory at d6000000 (64-bit, prefetchable) [size=8M]
Region 2: Memory at d6800000 (64-bit, prefetchable) [size=8M]
Region 4: Memory at d97d0000 (64-bit, prefetchable) [size=64K]
Expansion ROM at dc080000 [disabled] [size=512K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Product Name: QLogic 57800 10 Gigabit Ethernet
Read-only fields:
[PN] Part number: BCM957800
[MN] Manufacture ID: 31 30 32 38
[V0] Vendor specific: FFV15.00.14
[V1] Vendor specific: DSV1028VPDR.VER1.0
[V2] Vendor specific: NPY4
[V3] Vendor specific: PMT1367
[V4] Vendor specific: NMVQLogic
[V5] Vendor specific: DTINIC
[V6] Vendor specific: DCM1001009d7F1402009d7F2101009d7F2502009d7F32010089154301008915
[RV] Reserved: checksum good, 199 byte(s) reserved
End
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number b0-83-fe-ff-fe-cc-e2-ed
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [220 v1] #15
Kernel driver in use: bnx2x
Kernel modules: bnx2x

Spoiler: ethtool -i eno1

driver: bnx2x
version: 1.713.36-0 storm 7.13.11.0
firmware-version: FFV15.00.14 bc 7.14.12 phy 1.45
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Spoiler: pveversion -v

proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 14.2.6-pve1
ceph-fuse: 14.2.6-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 1.2.8-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-2
pve-container: 3.0-16
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

Any ideas??

 

[readyspace]

Hi, are you using cat 7 cables? If not, do try get cat 7 to see if it helps

 

[oz1cw7yymn]

I'm using Cat 6a with a length of <3m.
I have a cheap Buffalo 10GB NIC (LGY-PCIE-MG) in another machine, I had to do some kernel magic to enable it. It connects fine with 10GB, when I move the same cable to the x540 I get 1GB.
¯\_(ツ)_/¯

 

[psionic]

Hopefully not mixing 10G and 1G connections on 10G switch...

 

[oz1cw7yymn]

Good point - I had a cable connected to a 1GB switch for my initial switch configuration. However, no difference with only 10GB connections.

And it would be a very poor switch that couldn't negotiate 10GB for the x540 NIC when there was a 1GB cable connected. And even more strange that there is no problem doing 10GB when connected to the Buffalo NIC, even with both the x540 and a 1GB cable connected to the switch.

 

[vikozo]

is the switch port Auto Config or fixed to 10GB?

 

[psionic]

Good point - I had a cable connected to a 1GB switch for my initial switch configuration. However, no difference with only 10GB connections.

And it would be a very poor switch that couldn't negotiate 10GB for the x540 NIC when there was a 1GB cable connected. And even more strange that there is no problem doing 10GB when connected to the Buffalo NIC, even with both the x540 and a 1GB cable connected to the switch.

I guess the next question is what is the purpose of the 10G connection?

 

[oz1cw7yymn]

is the switch port Auto Config or fixed to 10GB?

Auto config - I haven't been able to set static 10GB, my options are auto config, auto-config 5GB and auto-config 2.5GB. They all auto configures to 1GB when connected to the X540.

I guess the next question is what is the purpose of the 10G connection?

The purpose of the 10GB connection is to connect the ceph cluster network for the recovery data. I have other machines with 10GB, I just can't get the X540 10GB card to actually work at 10GB.

 

[vikozo]

@James Pass
strange if you can't fix it to 10gig

 

[Stoiko Ivanov]

Dell R720x with an X540 NIC.

the X540 NICs I know are from Intel - not from broadcom

01:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57800 1/10 Gigabit Ethernet [14e4:168a] (rev 10)

hard to find concrete information on that - but this could be a 1G nic (or one which has 4 port - 2x1G copper, and 2x 10G SFP+ - see https://www.dell.com/downloads/global/products/pedge/poweredge_select_network_adapter.pdf)

* how is the card connected - copper/twisted pair, or via SFP+ module? (if it's a sfp+ module it could be that it's simply a 1G module and you need to change that)
* what's the output of:
** `ip link`
** `ethtool eno1` (replace eno1 with all NICs in your system (you get the names from the `ip link` command)

I hope this helps

 

[oz1cw7yymn]

The connections are all TP copper, no SFPs.

Spoiler

ethtool confirms 10GB for eno1 and eno2 but says no link even though I have a cable in eno1 connected to the 10GB switch.

Spoiler: ethtools

Settings for lo::
Link detected: yes
Settings for eno1::
Supported ports: [ TP ]
Supported link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 17
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: no
Settings for enp68s0f0::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
Settings for eno2::
Supported ports: [ TP ]
Supported link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 16
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: no
Settings for enp68s0f1::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
Settings for eno3::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 18
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: no
Settings for enp69s0f0::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
Settings for eno4::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 19
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: no
Settings for enp69s0f1::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
Settings for idrac::
Current message level: 0x00000007 (7)
drv probe link
Link detected: no

This is how I feel right now!

 

[Stoiko Ivanov]

ethtool confirms 10GB for eno1 and eno2 but says no link even though I have a cable in eno1 connected to the 10GB switch.

please post the complete output of `ip link` (inside code-tags)

is the link set up from the commandline:
`ip link set eno1 up`

I hope this helps

 

[oz1cw7yymn]

Man - doing ip link up switched it to 10GB! Sorry for bothering with such a simple problem in the end, I just thought that the NIC and the switch would be negotiating the physical link without needing the OS involved.

ip link before

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:cc:e2:ed brd ff:ff:ff:ff:ff:ff
3: enp68s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:96:67:d0 brd ff:ff:ff:ff:ff:ff
4: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:cc:e2:ef brd ff:ff:ff:ff:ff:ff
5: enp68s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:96:67:d1 brd ff:ff:ff:ff:ff:ff
6: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:cc:e2:f1 brd ff:ff:ff:ff:ff:ff
7: enp69s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:96:67:d4 brd ff:ff:ff:ff:ff:ff
8: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:cc:e2:f3 brd ff:ff:ff:ff:ff:ff
9: enp69s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:96:67:d5 brd ff:ff:ff:ff:ff:ff
10: idrac: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 5c:f9:dd:f8:2d:81 brd ff:ff:ff:ff:ff:ff

ethtool after

Settings for eno1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  100baseT/Full
                                         1000baseT/Full
                                         10000baseT/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 17
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Current message level: 0x00000000 (0)
                  
    Link detected: yes

Thanks again!

 

[spirit]

did you have
"auto eno1" in your /etc/network/interfaces ?