10G port not active

[musangberjanggut]

Hi Guys,

i have a problem. my host has 2x 1G port running on igb driver and 2x 100/1000/10G port running on ixgbe driver. however, i can see that currently my 10g port is not functioning and not active. even on the ports i can see the light is on, i can see inside the OS it is not functioning. please find below are the output.

mii-tool
eth0: negotiated 1000baseT-FD flow-control, link ok
eth1: negotiated 1000baseT-FD flow-control, link ok
SIOCGMIIPHY on 'eth2' failed: Operation not supported
SIOCGMIIPHY on 'eth3' failed: Operation not supported

-------------------------------------------------------

ethtool -i eth2
driver: ixgbe
version: 3.21.2
firmware-version: 0x80000314, 1.380.0
bus-info: 0000:82:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

------------------------------------------------

ethtool -i eth2
driver: ixgbe
version: 3.21.2
firmware-version: 0x80000314, 1.380.0
bus-info: 0000:82:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
root@cloudhost05:~# ethtool eth2
Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: external
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: no

if you can see, ethtool also show that the link is not detected.

below are the output of lspci -kv

--------------------------------------

82:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
Subsystem: QUANTA Computer Inc Device 8999
Flags: bus master, fast devsel, latency 0, IRQ 56
Memory at fb600000 (64-bit, prefetchable) [size=2M]
I/O ports at f020
Memory at fb804000 (64-bit, prefetchable) [size=16K]
Expansion ROM at fb980000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1d0] Access Control Services
Kernel driver in use: ixgbe

82:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
Subsystem: QUANTA Computer Inc Device 8999
Flags: bus master, fast devsel, latency 0, IRQ 60
Memory at fb400000 (64-bit, prefetchable) [size=2M]
I/O ports at f000
Memory at fb800000 (64-bit, prefetchable) [size=16K]
Expansion ROM at fb900000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1d0] Access Control Services
Kernel driver in use: ixgbe

-----------------------------------------------

my pveversion as per below:

pveversion -v
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
pve-manager: 3.3-1 (running version: 3.3-1/a06c9f73)
pve-kernel-2.6.32-32-pve: 2.6.32-136
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-34
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-5
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

-------------------------------

please advise how can we troubleshoot this kind of issue. do i need to upgrade my ixgbe driver? looking forward for your reply.

thanks.

 

[NStorm]

Its definitely not a proxmox specific problem.
First, check your cabling. 10GBase-T specifies it should be at least Cat.6 cable with a maximum distance of 100m on Cat 6A (better than just Cat.6) cable.
Second, do you have 10Gb green light on card? Because ethtool tells the link is negotiated at 1Gbps: Speed: 1000Mb/s Do you connect to a 10GBase-T switch or directly NIC to NIC?
I've had similar problem with exactly the the same Intel X540-T2 cards. But mine connected at least, only the speed was always only 1Gbps, never connected at 10Gbps. I've tried different drivers, different distros and even different PCs. Last time I've tried it on Windows Server OS, and the driver settings didn't even had 10Gbps speed settings. I've contacted Intel on this and they've confirmed my cards were defective and should be replaced. Something with firmware/nvram I guess. I've returned them to a supplier and never dealt with 10GBase-T again. Cheap Asus/D-Link SFP+ cards with copper Direct Attach cable works like a charm when you don't need more that 10 meters connections.

 

[musangberjanggut]

i believe it should support 1000Mb/s port. currently we connect using cat 6 to 1Gb switch and it is just 5 meter length. the lights also turn on. hopefully the cards is faulty so easy for me to call the supplier and get an rma/fix it. anyway, thanks for the updates bro.

 

[NStorm]

Oh, I thought you are trying to setup it as 10Gbps based on topic. Yes, it should support 1Gbit though. Didn't you did 'ifconfig eth2 up'? Because most of the cards drivers don't "wake up" and doesn't use the link if the interface is disabled. You should try to configure an IP on it and try a ping test.

 

[musangberjanggut]

already tried. but seems like no luck. any other options.

 

[NStorm]

Not really, except for try other PC/OS.

 

[kurniawanep]

recommended cat7 its save