[SOLVED] Need help getting SFP+ to link (ixgbe)

Jul 18, 2018
14
0
6
40
I have a Dell server with an XA520-DA1 installed. It works fine with 1 Gb SFPs. I bought some new SFPs from fs.com, and I can't get the link to come up.

My Mikrotik shows good light levels, and I don't see any errors in proxmox. What should I check for?

Code:
root@proxmox01:~# lspci | grep Ethernet
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
82:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

root@proxmox01:~# dmesg | grep ixgbe
[    1.818514] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[    1.818515] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    1.988221] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 24, Tx Queue count = 24 XDP Queue count = 0
[    1.988347] ixgbe 0000:82:00.0: PCI Express bandwidth of 32GT/s available
[    1.988348] ixgbe 0000:82:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    1.988469] ixgbe 0000:82:00.0: MAC: 2, PHY: 20, SFP+: 5, PBA No: Unknown
[    1.988471] ixgbe 0000:82:00.0: 00:1b:21:bc:a9:b6
[    1.989600] ixgbe 0000:82:00.0: Intel(R) 10 Gigabit Network Connection
[    1.990384] ixgbe 0000:82:00.0 enp130s0: renamed from eth1
[    6.582170] ixgbe 0000:82:00.0: registered PHC device on enp130s0
[    6.764070] ixgbe 0000:82:00.0 enp130s0: detected SFP+: 5
[  480.767835] ixgbe 0000:82:00.0: removed PHC on enp130s0
[  481.402374] ixgbe 0000:82:00.0: registered PHC device on enp130s0
[  481.580042] ixgbe 0000:82:00.0 enp130s0: detected SFP+: 5

root@proxmox01:~# uname -r
4.15.18-12-pve

Code:
root@proxmox01:~# ip addr show dev enp130s0
3: enp130s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1001 state DOWN group default qlen 1000
    link/ether 00:1b:21:bc:a9:b6 brd ff:ff:ff:ff:ff:ff

Code:
root@proxmox01:~# lspci -vv | grep -A64 82:00.0
82:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-1
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 66
        NUMA node: 1
        Region 0: Memory at c8200000 (64-bit, non-prefetchable) [size=512K]
        Region 2: I/O ports at 8000 [size=32]
        Region 4: Memory at c8280000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00002000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us, L1 <8us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [e0] Vital Product Data
                Not readable
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn+ ChkCap+ ChkEn+
        Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-bc-a9-b6
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 128, stride: 2, Device ID: 10ed
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000c8284000 (64-bit, non-prefetchable)
                Region 3: Memory at 00000000c8384000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
 
Last edited:
Hi,

not all SFP modules are programmed (compatible) for any dev.
 
I'm not 100% sure what that means, but I think you might be saying the SFP is not compatible with the X520.

If so, that's an easy fix. I can start an RMA with fs.com. I specifically bought an INTEL compatible sfp, so maybe they sent me the wrong one.
 
I'm not 100% sure what that means, but I think you might be saying the SFP is not compatible with the X520.
Yes this is correct.

But this is only a guess.
The SPF module has nothing to do with the driver.
 
Yes this is correct.

But this is only a guess.
The SPF module has nothing to do with the driver.

Just commenting in case some other poor soul needs the resource. Wolfgang is 100% correct, I bought an Intel branded sfp off ebay (for cheaper than fs.com) and it came right up.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!