[SOLVED] Need help getting SFP+ to link (ixgbe)

[Colby Westerfield]

I have a Dell server with an XA520-DA1 installed. It works fine with 1 Gb SFPs. I bought some new SFPs from fs.com, and I can't get the link to come up.

My Mikrotik shows good light levels, and I don't see any errors in proxmox. What should I check for?

root@proxmox01:~# lspci | grep Ethernet
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
82:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

root@proxmox01:~# dmesg | grep ixgbe
[    1.818514] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[    1.818515] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    1.988221] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 24, Tx Queue count = 24 XDP Queue count = 0
[    1.988347] ixgbe 0000:82:00.0: PCI Express bandwidth of 32GT/s available
[    1.988348] ixgbe 0000:82:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    1.988469] ixgbe 0000:82:00.0: MAC: 2, PHY: 20, SFP+: 5, PBA No: Unknown
[    1.988471] ixgbe 0000:82:00.0: 00:1b:21:bc:a9:b6
[    1.989600] ixgbe 0000:82:00.0: Intel(R) 10 Gigabit Network Connection
[    1.990384] ixgbe 0000:82:00.0 enp130s0: renamed from eth1
[    6.582170] ixgbe 0000:82:00.0: registered PHC device on enp130s0
[    6.764070] ixgbe 0000:82:00.0 enp130s0: detected SFP+: 5
[  480.767835] ixgbe 0000:82:00.0: removed PHC on enp130s0
[  481.402374] ixgbe 0000:82:00.0: registered PHC device on enp130s0
[  481.580042] ixgbe 0000:82:00.0 enp130s0: detected SFP+: 5

root@proxmox01:~# uname -r
4.15.18-12-pve

root@proxmox01:~# ip addr show dev enp130s0
3: enp130s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1001 state DOWN group default qlen 1000
    link/ether 00:1b:21:bc:a9:b6 brd ff:ff:ff:ff:ff:ff

root@proxmox01:~# lspci -vv | grep -A64 82:00.0
82:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-1
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 66
        NUMA node: 1
        Region 0: Memory at c8200000 (64-bit, non-prefetchable) [size=512K]
        Region 2: I/O ports at 8000 [size=32]
        Region 4: Memory at c8280000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00002000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us, L1 <8us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [e0] Vital Product Data
                Not readable
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn+ ChkCap+ ChkEn+
        Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-bc-a9-b6
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 128, stride: 2, Device ID: 10ed
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000c8284000 (64-bit, non-prefetchable)
                Region 3: Memory at 00000000c8384000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

 

[wolfgang]

Hi,

not all SFP modules are programmed (compatible) for any dev.

 

[Colby Westerfield]

I'm not 100% sure what that means, but I think you might be saying the SFP is not compatible with the X520.

If so, that's an easy fix. I can start an RMA with fs.com. I specifically bought an INTEL compatible sfp, so maybe they sent me the wrong one.

 

[wolfgang]

I'm not 100% sure what that means, but I think you might be saying the SFP is not compatible with the X520.

Yes this is correct.

But this is only a guess.
The SPF module has nothing to do with the driver.

 

[Colby Westerfield]

Yes this is correct.

But this is only a guess.
The SPF module has nothing to do with the driver.

Just commenting in case some other poor soul needs the resource. Wolfgang is 100% correct, I bought an Intel branded sfp off ebay (for cheaper than fs.com) and it came right up.