Proxmox 6.1 Intel X-520-T2 not getting detected

[Haider Jarral]

Hello Experts,

I have been struggling with this Intel X-520-T2 card, I have tried everything bit it just fails to show up, please if anyone can provide some direction or has seen similar issue using this card.

lshw -c network -businfo
Bus info          Device      Class          Description
========================================================
pci@0000:05:00.0  eno1        network        I350 Gigabit Network Connection
pci@0000:05:00.1  eno2        network        I350 Gigabit Network Connection
pci@0000:05:00.2  eno3        network        I350 Gigabit Network Connection
pci@0000:05:00.3  eno4        network        I350 Gigabit Network Connection
pci@0000:84:00.0              network        Intel Corporation
pci@0000:84:00.1              network        Intel Corporation
                  vmbr0       network        Ethernet interface
                  bond0       network        Ethernet interface

 

[Haider Jarral]

Relevant outputs.

 

[Haider Jarral]

Anyone who has used Intel 520-T2 card on proxmox ?

https://ark.intel.com/content/www/u...hernet-converged-network-adapter-x520-t2.html

 

[Stoiko Ivanov]

check the journal of the last boot for messages from the NIC driver (afaik it should be ixgbe for the X520)

also please post the output of `lscpi -nnk`

I hope this helps!

 

[Haider Jarral]

Here is the requested output,

~#  lspci -k | sed -n '/Ethernet/,/Kernel/p'
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    Subsystem: Super Micro Computer Inc I350 Gigabit Network Connection
    Kernel driver in use: igb
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    Subsystem: Super Micro Computer Inc I350 Gigabit Network Connection
    Kernel driver in use: igb
05:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    Subsystem: Super Micro Computer Inc I350 Gigabit Network Connection
    Kernel driver in use: igb
05:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    Subsystem: Super Micro Computer Inc I350 Gigabit Network Connection
    Kernel driver in use: igb
84:00.0 Ethernet controller: Intel Corporation Device 0001 (rev 01)
    Subsystem: Device 1b52:151c
84:00.1 Ethernet controller: Intel Corporation Device 0001 (rev 01)
    Subsystem: Device 1b52:151c

 

[Stoiko Ivanov]

did you check the journal?

please provide the output of `lspci -nnk` (without sed or adapting the cli-parameters)

Thanks!

 

[Haider Jarral]

Here is full output, I am sorry I am not getting the which journal should I check.

 

[Haider Jarral]

Here you go

 

[Stoiko Ivanov]

Hmm - the journal would not indicate that the driver got loaded (same goes for the lspci output)

* you could try to boot the new 5.4 kernel (install pve-kernel-5.4) - maybe it's just a new model and the pci-ids did not get updated yet
* you can load both intel 10g nic-drivers and check dmesg for potential informations:

modprobe ixgbe
dmesg
modprobe i40e
dmesg

I hope this helps!

 

[Haider Jarral]

Same thing in pve-kernel-5.4

after bootup, when I issue modprobe ixgbe

only these new entries appear in dmesg

[   81.471183] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[   81.471184] ixgbe: Copyright (c) 1999-2016 Intel Corporation.

However If I manually bind it

echo 8086 0001 > /sys/bus/pci/drivers/ixgbe/new_id

I see these new entries in dmesg

[   81.471183] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[   81.471184] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[  193.760637] ixgbe 0000:84:00.0: Warning firmware error detected FWSM: 0x00000000
[  193.784354] ixgbe 0000:84:00.0: Failed to allocate MSI-X interrupts. Err: -34
[  193.784356] ixgbe 0000:84:00.0: Disabling SR-IOV support
[  193.784357] ixgbe 0000:84:00.0: Disabling RSS support
[  193.784403] ixgbe 0000:84:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[  193.784779] ixgbe 0000:84:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[  193.784861] ixgbe 0000:84:00.0: MAC: 1, PHY: 2, PBA No: E95990-004
[  193.784863] ixgbe 0000:84:00.0: 00:10:04:01:5c:77
[  193.786532] ixgbe 0000:84:00.0: Intel(R) 10 Gigabit Network Connection
[  193.786560] libphy: ixgbe-mdio: probed
[  193.803326] ixgbe 0000:84:00.0 enp132s0f0: renamed from eth0
[  194.172638] ixgbe 0000:84:00.1: Warning firmware error detected FWSM: 0x00000000
[  194.196332] ixgbe 0000:84:00.1: Failed to allocate MSI-X interrupts. Err: -34
[  194.196333] ixgbe 0000:84:00.1: Disabling SR-IOV support
[  194.196335] ixgbe 0000:84:00.1: Disabling RSS support
[  194.196396] ixgbe 0000:84:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[  194.196772] ixgbe 0000:84:00.1: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[  194.196856] ixgbe 0000:84:00.1: MAC: 1, PHY: 2, PBA No: E95990-004
[  194.196858] ixgbe 0000:84:00.1: 00:10:04:01:5c:76
[  194.198557] ixgbe 0000:84:00.1: Intel(R) 10 Gigabit Network Connection
[  194.198587] libphy: ixgbe-mdio: probed
[  194.199130] ixgbe 0000:84:00.1 enp132s0f1: renamed from eth0

Now they do show up but link shows down

 

[Stoiko Ivanov]

what's the output of 'ip link'
did you enable the nic with `ip link set enp132s0f1 up` ?

Warning firmware error detected FWSM: 0x00000000

maybe try to upgrade the firmware of the nic if available
else - maybe the NIC is broken?

I hope this helps!

 

[Haider Jarral]

The problem is consistent on all 4 servers having same NIC card, likelihood of all NIC cards broken is less, I did try installing the firmware didn't help.

 

[Haider Jarral]

Ok the ports came up, but it not consistent across reboot, how can make it consistent or run theses commands everytime after boot automatically

root@voip:~# modprobe ixgbe
root@voip:~# echo 8086 0001 > /sys/bus/pci/drivers/ixgbe/new_id
root@voip:~# ip link set enp132s0f1 up
root@voip:~# ip link set enp132s0f0 up
root@voip:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:e9:66:f4 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:e9:66:f4 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:e9:66:f6 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:e9:66:f7 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:e9:66:f4 brd ff:ff:ff:ff:ff:ff
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:e9:66:f4 brd ff:ff:ff:ff:ff:ff
8: enp132s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:10:04:01:5c:77 brd ff:ff:ff:ff:ff:ff
9: enp132s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:10:04:01:5c:76 brd ff:ff:ff:ff:ff:ff
root@voip:~#

 

[Stoiko Ivanov]

does traffic come in/out over the NIC?

you can try to add the 'echo' as pre-up command in /etc/network/interfaces - it's still odd that the driver does not bind to the card(s)

 

[Haider Jarral]

nevermind it failed with its original intel firmware, and with default include firmware it kept on flapping, I think I will be replacing this card, I don't think there is any other option.

[   46.733756] Intel(R) 10GbE PCI Express Linux Network Driver - version 5.6.5
[   46.733759] Copyright(c) 1999 - 2019 Intel Corporation.
[   51.085263] ixgbe 0000:84:00.0 0000:84:00.0 (uninitialized): init_shared_code failed: -11
[   51.085440] ixgbe: probe of 0000:84:00.0 failed with error -11
[   51.085529] ixgbe 0000:84:00.1 0000:84:00.1 (uninitialized): init_shared_code failed: -11
[   51.085678] ixgbe: probe of 0000:84:00.1 failed with error -11

 

[Stoiko Ivanov]

sounds like a good plan!

 

[Haider Jarral]

Thank you for all your assistance and time.

 

[Stoiko Ivanov]

glad to be of help!

let us know if replacing the cards helped.
Thanks!

 

[dthompson]

I had this issue with a previous version and was able to get it running with the following. This is copied from my previous post here:

This is what I did to get around the issue:

1.) cd /tmp
2.) Download the link from above (also here)
3.) tar -xvpf ixgbe-5.7.1.tar.gz
4.) cd ixgbe-5.7.1/src
5.) apt install pve-headers build-essential
6.) make install

As per the read me notes, the driver gets installed in the following location:
/lib/modules/5.4.44-2-pve/updates/drivers/net/ethernet/intel/ixgbe

7.) modinfo ixgbe
8.) modprobe ixgbe [parameter=port1_value,port2_value]
9.) rmmod ixgbe; modprobe ixgbe
10.) ifdown <NETWORK Interface>

In my case, it was the following on one of my servers:
ifdown enp3s0f0
then
ifup enp3s0f0

Repeat for your other Nic(s) that are also having the same issue

11.) update-initramfs -u

Restart your server and you "should" be good to go!

 

[Jarod42]

Thank you for all your assistance and time.

I know it's been some time, but did you manage to get the cards working correctly?
I ask because I'm facing the exact same problem and @dthompson 's solution did not work.

I got in contact with Intel and they told me that according to their database this is PCI error. The most obvious sign is wrong sub-vendor id. I've tried the card in 2 different mainboards. First one was an ASUS B450 Plus and the second a Supermicro X11SAE.
I would have expected to get the card working with the server board at least...

Any help is greatly appreciated.

Jarod