Intel X520 won't work in Proxmox

Luna · Jan 8, 2017

Hi,

I migrated to Proxmox because I wanted to get rid of ESXi, but I can't get the Intel X520-DA2 card in it working. It does not make sense to me since the card is recognized by Debian, but just won't connect when I put the network cable in it.

Can someone please help me with getting this card running? I am rather new to Linux, so maybe I did something stupid.

Some information:

Code:

root@x:~# pveversion
pve-manager/4.4-5/c43015a5 (running kernel: 4.4.35-1-pve)

Code:

root@x:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
  valid_lft forever preferred_lft forever
  inet6 ::1/128 scope host
  valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  link/ether 90:e2:ba:5e:78:48 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  link/ether 90:e2:ba:5e:78:49 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a4 brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a5 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a6 brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a7 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a4 brd ff:ff:ff:ff:ff:ff
  inet 10.0.2.1/16 brd 10.0.255.255 scope global vmbr0
  valid_lft forever preferred_lft forever
  inet6 fe80::ec4:7aff:fe08:a2a4/64 scope link
  valid_lft forever preferred_lft forever
9: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a5 brd ff:ff:ff:ff:ff:ff
  inet 10.1.2.1/16 brd 10.1.255.255 scope global vmbr1
  valid_lft forever preferred_lft forever
  inet6 fe80::ec4:7aff:fe08:a2a5/64 scope link
  valid_lft forever preferred_lft forever
10: zt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast state UNKNOWN group default qlen 1000
  link/ether 96:c3:30:87:15:af brd ff:ff:ff:ff:ff:ff
  inet 10.100.0.10/16 brd 10.100.255.255 scope global zt0
  valid_lft forever preferred_lft forever
  inet6 fde5:cd7a:9e1c:c8ce:9599:930d:f89b:8bd5/88 scope global
  valid_lft forever preferred_lft forever
  inet6 fe80::94c3:30ff:fe87:15af/64 scope link
  valid_lft forever preferred_lft forever
11: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
  link/ether 22:02:b3:1a:72:e7 brd ff:ff:ff:ff:ff:ff
12: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
  link/ether 5a:03:33:6a:84:d8 brd ff:ff:ff:ff:ff:ff
13: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
  link/ether 52:3c:ed:73:72:d7 brd ff:ff:ff:ff:ff:ff
14: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
  link/ether 5a:34:a9:e8:b4:7a brd ff:ff:ff:ff:ff:ff

Code:

root@x:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface eth1 inet manual

iface eth0 inet manual

iface eth2 inet manual

iface eth3 inet manual

iface eth4 inet manual

iface eth5 inet manual

auto vmbr0
iface vmbr0 inet static
  address  10.0.2.1
  netmask  255.255.0.0
  gateway  10.0.0.1
  bridge_ports eth0
  bridge_stp off
  bridge_fd 0

auto vmbr1
iface vmbr1 inet static
  address  10.1.2.1
  netmask  255.255.0.0
  gateway  10.1.0.1
  bridge_ports eth2
  bridge_stp off
  bridge_fd 0

eth0 = Intel X520-DA2 10 Gb/s
eth1 = Intel i354 1 Gb/s
eth2 = Intel i354 1 Gb/s
eth3 = Intel X520-DA2 10 Gb/s
eth4 = Intel i354 1 Gb/s
eth5 = Intel i354 1 Gb/s

I used eth1 for the vmbr0 bridge in my testenvironment (where I don't have 10 Gb/s switches), that worked perfectly. In the primary rack I connected the Intel X520 to the switch and nothing happened. Another cable from another host (which works) also does not work and the intended cable from this host works on other hosts.

What did I do wrong?

A lot of thanks in advance!

Luna · Jan 8, 2017

I also came accross some programs/commands in Linux that might give some more information.

Code:

root@x:~# lspci -v
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  Subsystem: Intel Corporation Ethernet Server Adapter X520-2
  Flags: bus master, fast devsel, latency 0, IRQ 22
  Memory at 80000000 (64-bit, non-prefetchable) [size=512K]
  I/O ports at c020 [size=32]
  Memory at 80100000 (64-bit, non-prefetchable) [size=16K]
  Capabilities: [40] Power Management version 3
  Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
  Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
  Capabilities: [a0] Express Endpoint, MSI 00
  Capabilities: [100] Advanced Error Reporting
  Capabilities: [140] Device Serial Number 90-e2-ba-ff-ff-5e-78-48
  Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
  Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
  Kernel driver in use: ixgbe

04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  Subsystem: Intel Corporation Ethernet Server Adapter X520-2
  Flags: bus master, fast devsel, latency 0, IRQ 23
  Memory at 80080000 (64-bit, non-prefetchable) [size=512K]
  I/O ports at c000 [size=32]
  Memory at 80304000 (64-bit, non-prefetchable) [size=16K]
  Capabilities: [40] Power Management version 3
  Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
  Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
  Capabilities: [a0] Express Endpoint, MSI 00
  Capabilities: [100] Advanced Error Reporting
  Capabilities: [140] Device Serial Number 90-e2-ba-ff-ff-5e-78-48
  Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
  Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
  Kernel driver in use: ixgbe

Code:

root@x:~# dmesg | grep Gigabit
[  1.239616] Intel(R) Gigabit Ethernet Network Driver - version 5.3.5.3
[  1.418185] ixgbe 0000:04:00.0 eth0: Intel(R) 10 Gigabit Network Connection
[  2.566203] ixgbe 0000:04:00.1 eth1: Intel(R) 10 Gigabit Network Connection
[  2.621489] igb 0000:00:14.0: Intel(R) Gigabit Ethernet Network Connection
[  4.009299] igb 0000:00:14.1: Intel(R) Gigabit Ethernet Network Connection
[  5.397285] igb 0000:00:14.2: Intel(R) Gigabit Ethernet Network Connection
[  6.805259] igb 0000:00:14.3: Intel(R) Gigabit Ethernet Network Connection

Code:

root@x:~# brctl show
bridge name  bridge id  STP enabled  interfaces
vmbr0  8000.0cc47a08a2a4  no  eth1
  tap101i0
  tap102i0
  tap103i0
vmbr1  8000.0cc47a08a2a5  no  eth2
  tap100i0

If anyone needs more, let me know

.

manu · Jan 10, 2017

Hi Luna

Please enter the following commands:

lspci -k | sed -n '/Ethernet/,/Kernel/p'

this should tell you for which NIC, which driver is the kernel using
and post the result here

on my system it looks like this:
04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
Subsystem: Super Micro Computer Inc Device 1521
Kernel driver in use: igb
04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
Subsystem: Super Micro Computer Inc Device 1521
Kernel driver in use: igb

Hopefully for each of your device you should see a kernel driver in use.

For each device where the kernel has the corresponding driver, the kernel will create a 'eth' * name.

You can list the eth* devices with

ip link

if you see the string LOWER_UP if means that the driver is active and detected an Ethernet network.
If you see that LOWER_UP flg you're good to configure an ip adress on the ethernet device.

Luna · Jan 10, 2017

Hi Manu,

Thanks for your reply

. Below are the outputs:

Code:

root@x:~# lspci -k | sed -n '/Ethernet/ ,/Kernel/p'
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
  Subsystem: Super Micro Computer Inc Device 1f41
  Kernel driver in use: igb
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
  Subsystem: Super Micro Computer Inc Device 1f41
  Kernel driver in use: igb
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
  Subsystem: Super Micro Computer Inc Device 1f41
  Kernel driver in use: igb
00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
  Subsystem: Super Micro Computer Inc Device 1f41
  Kernel driver in use: igb
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  Subsystem: Intel Corporation Ethernet Server Adapter X520-2
  Kernel driver in use: ixgbe
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  Subsystem: Intel Corporation Ethernet Server Adapter X520-2
  Kernel driver in use: ixgbe

Code:

root@x:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
  link/ether 90:e2:ba:5e:78:48 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
  link/ether 90:e2:ba:5e:78:49 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a4 brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a5 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a6 brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a7 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a4 brd ff:ff:ff:ff:ff:ff
9: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
  link/ether 0c:c4:7a:08:a2:a5 brd ff:ff:ff:ff:ff:ff
10: zt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
  link/ether 96:c3:30:87:15:af brd ff:ff:ff:ff:ff:ff
11: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
  link/ether 22:02:b3:1a:72:e7 brd ff:ff:ff:ff:ff:ff
12: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN mode DEFAULT group default qlen 1000
  link/ether 5a:03:33:6a:84:d8 brd ff:ff:ff:ff:ff:ff
13: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
  link/ether 52:3c:ed:73:72:d7 brd ff:ff:ff:ff:ff:ff
14: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
  link/ether 5a:34:a9:e8:b4:7a brd ff:ff:ff:ff:ff:ff

eth0 and eth3 (the 10 GBe NIC's) don't have the LOWER_UP string right? Is that the problem?

Thanks!

manu · Jan 10, 2017

yes you should have the LOWER UP here. Check your cable connectivitiy.
the pve kernel includes the ixgbe driver in version 4.4.6 which has support for this device

now there are some hints related to the connector in the driver readme maybe you should check this too http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005688.html

Luna · Jan 10, 2017

Hi Manu,

Thanks for your quick reply! I will put the cables back in (which did not respond before, the LED didn't blink) when I get home too see if that changes the string to LOWER_UP.

Luna · Jan 10, 2017

Hi Manu,

I put different cables in the NIC, but nothing happened. The lights on the switch, nor the lights of the NIC went blinking. Also the state stayed the same, so no 'LOWER_UP' on the 10 gigabit interface. The weird thing is that ESXi had no trouble with this NICs.

What can I try now to troubleshoot it?

tez · Jan 14, 2017

I would suggest looking at https://sourceforge.net/p/e1000/bugs/399/ - this sounds like something that I hit several years ago and specifying the module parameter detailed in the report may fix the problem you are seeing.

Luna · Jan 14, 2017

tez said:
I would suggest looking at https://sourceforge.net/p/e1000/bugs/399/ - this sounds like something that I hit several years ago and specifying the module parameter detailed in the report may fix the problem you are seeing.

Hi Tez. I do have Intel SFP+ modules, can this bug then also apply to my situation? I read a lot about non-Intel modules.

Thanks!

Search

Search

Intel X520 won't work in Proxmox

Luna

New Member

Luna

New Member

manu

Proxmox Staff Member

Luna

New Member

manu

Proxmox Staff Member

Luna

New Member

Luna

New Member

tez

Member

Luna

New Member