[SOLVED] 10GB NIC only works in 1GB mode

O

oz1cw7yymn

Well-Known Member
Feb 13, 2019
93
12
48
Hi,

I am installing a Dell R720x with an X540 NIC. This card has two 10GB and 2 1GB, however they are all just working at 1GB. When I connect the card to a 10GB switch it negotiates 1GB only. I can't see any relevant issues in dmesg and I don't know if there is a way to disable/enable 10GB capability somehow. The kernel is using the bnx2x driver.

Here are the relevant outputs:
[ +0.216486] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.713.36-0 (2014/02/10)
[ +0.000278] bnx2x 0000:01:00.0: msix capability found
[ +0.000559] bnx2x 0000:01:00.0: part number 394D4342-30383735-30305430-473030
[ +0.094200] bnx2x 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ +0.000125] bnx2x 0000:01:00.1: msix capability found
[ +0.000371] bnx2x 0000:01:00.1: part number 394D4342-30383735-30305430-473030
[ +0.010540] bnx2x 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ +0.000083] bnx2x 0000:01:00.2: msix capability found
[ +0.000366] bnx2x 0000:01:00.2: part number 394D4342-30383735-30305430-473030
[ +0.057505] bnx2x 0000:01:00.2: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ +0.000120] bnx2x 0000:01:00.3: msix capability found
[ +0.000377] bnx2x 0000:01:00.3: part number 394D4342-30383735-30305430-473030
[ +0.005917] bnx2x 0000:01:00.3: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ +0.003137] bnx2x 0000:01:00.0 eno1: renamed from eth0
[ +0.020478] bnx2x 0000:01:00.1 eno2: renamed from eth2
[ +0.064607] bnx2x 0000:01:00.2 eno3: renamed from eth4
[ +0.014504] bnx2x 0000:01:00.3 eno4: renamed from eth6

01:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57800 1/10 Gigabit Ethernet [14e4:168a] (rev 10)
Subsystem: Dell BCM57800 10-Gigabit Ethernet [1028:1f5d]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 64
NUMA node: 0
Region 0: Memory at d6000000 (64-bit, prefetchable) [size=8M]
Region 2: Memory at d6800000 (64-bit, prefetchable) [size=8M]
Region 4: Memory at d97d0000 (64-bit, prefetchable) [size=64K]
Expansion ROM at dc080000 [disabled] [size=512K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Product Name: QLogic 57800 10 Gigabit Ethernet
Read-only fields:
[PN] Part number: BCM957800
[MN] Manufacture ID: 31 30 32 38
[V0] Vendor specific: FFV15.00.14
[V1] Vendor specific: DSV1028VPDR.VER1.0
[V2] Vendor specific: NPY4
[V3] Vendor specific: PMT1367
[V4] Vendor specific: NMVQLogic
[V5] Vendor specific: DTINIC
[V6] Vendor specific: DCM1001009d7F1402009d7F2101009d7F2502009d7F32010089154301008915
[RV] Reserved: checksum good, 199 byte(s) reserved
End
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number b0-83-fe-ff-fe-cc-e2-ed
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [220 v1] #15
Kernel driver in use: bnx2x
Kernel modules: bnx2x

driver: bnx2x
version: 1.713.36-0 storm 7.13.11.0
firmware-version: FFV15.00.14 bc 7.14.12 phy 1.45
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 14.2.6-pve1
ceph-fuse: 14.2.6-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 1.2.8-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-2
pve-container: 3.0-16
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

Any ideas??
 
I'm using Cat 6a with a length of <3m.
I have a cheap Buffalo 10GB NIC (LGY-PCIE-MG) in another machine, I had to do some kernel magic to enable it. It connects fine with 10GB, when I move the same cable to the x540 I get 1GB.
¯\_(ツ)_/¯
 
Good point - I had a cable connected to a 1GB switch for my initial switch configuration. However, no difference with only 10GB connections.

And it would be a very poor switch that couldn't negotiate 10GB for the x540 NIC when there was a 1GB cable connected. And even more strange that there is no problem doing 10GB when connected to the Buffalo NIC, even with both the x540 and a 1GB cable connected to the switch.
 
oz1cw7yymn said:
Good point - I had a cable connected to a 1GB switch for my initial switch configuration. However, no difference with only 10GB connections.

And it would be a very poor switch that couldn't negotiate 10GB for the x540 NIC when there was a 1GB cable connected. And even more strange that there is no problem doing 10GB when connected to the Buffalo NIC, even with both the x540 and a 1GB cable connected to the switch.
Click to expand...
I guess the next question is what is the purpose of the 10G connection?
 
vikozo said:
is the switch port Auto Config or fixed to 10GB?
Click to expand...

Auto config - I haven't been able to set static 10GB, my options are auto config, auto-config 5GB and auto-config 2.5GB. They all auto configures to 1GB when connected to the X540.

James Pass said:
I guess the next question is what is the purpose of the 10G connection?
Click to expand...
The purpose of the 10GB connection is to connect the ceph cluster network for the recovery data. I have other machines with 10GB, I just can't get the X540 10GB card to actually work at 10GB.
 
oz1cw7yymn said:
Dell R720x with an X540 NIC.
Click to expand...
the X540 NICs I know are from Intel - not from broadcom
oz1cw7yymn said:
01:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57800 1/10 Gigabit Ethernet [14e4:168a] (rev 10)
Click to expand...
hard to find concrete information on that - but this could be a 1G nic (or one which has 4 port - 2x1G copper, and 2x 10G SFP+ - see https://www.dell.com/downloads/global/products/pedge/poweredge_select_network_adapter.pdf)

* how is the card connected - copper/twisted pair, or via SFP+ module? (if it's a sfp+ module it could be that it's simply a 1G module and you need to change that)
* what's the output of:
** `ip link`
** `ethtool eno1` (replace eno1 with all NICs in your system (you get the names from the `ip link` command)

I hope this helps
 
The connections are all TP copper, no SFPs.
ports.jpg

ethtool confirms 10GB for eno1 and eno2 but says no link even though I have a cable in eno1 connected to the 10GB switch.


Settings for lo::
Link detected: yes
Settings for eno1::
Supported ports: [ TP ]
Supported link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 17
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: no
Settings for enp68s0f0::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
Settings for eno2::
Supported ports: [ TP ]
Supported link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Half 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 16
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: no
Settings for enp68s0f1::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
Settings for eno3::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 18
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: no
Settings for enp69s0f0::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
Settings for eno4::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 19
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: no
Settings for enp69s0f1::
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
Settings for idrac::
Current message level: 0x00000007 (7)
drv probe link
Link detected: no


This is how I feel right now!
networking_problems_2x.png
 

Attachments

  • ports.jpg
    ports.jpg
    428 KB · Views: 108
oz1cw7yymn said:
ethtool confirms 10GB for eno1 and eno2 but says no link even though I have a cable in eno1 connected to the 10GB switch.
Click to expand...
please post the complete output of `ip link` (inside code-tags)

is the link set up from the commandline:
`ip link set eno1 up`

I hope this helps
 
Man - doing ip link up switched it to 10GB! Sorry for bothering with such a simple problem in the end, I just thought that the NIC and the switch would be negotiating the physical link without needing the OS involved.

ip link before
Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:cc:e2:ed brd ff:ff:ff:ff:ff:ff
3: enp68s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:96:67:d0 brd ff:ff:ff:ff:ff:ff
4: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:cc:e2:ef brd ff:ff:ff:ff:ff:ff
5: enp68s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:96:67:d1 brd ff:ff:ff:ff:ff:ff
6: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:cc:e2:f1 brd ff:ff:ff:ff:ff:ff
7: enp69s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:96:67:d4 brd ff:ff:ff:ff:ff:ff
8: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:cc:e2:f3 brd ff:ff:ff:ff:ff:ff
9: enp69s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:96:67:d5 brd ff:ff:ff:ff:ff:ff
10: idrac: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 5c:f9:dd:f8:2d:81 brd ff:ff:ff:ff:ff:ff

ethtool after
Code: 
Settings for eno1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  100baseT/Full
                                         1000baseT/Full
                                         10000baseT/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 17
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Current message level: 0x00000000 (0)
                  
    Link detected: yes

Thanks again!
 
  • Like
Reactions: Stoiko Ivanov
You must log in or register to reply here.
Top Bottom