Intel X540-T2 "adapter removed"

[proxwolfe]

So I am trying to build a new cluster with 10gbe enabled nodes using intel X540-T2 cards.

The first node is up and running apparently fine.

The second node is acting up: I see this in dmesg during boot (and the X540-T2 does not show up in the gui):

dmesg | grep ixgbe [ 3.541917] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver [ 3.541924] ixgbe: Copyright (c) 1999-2016 Intel Corporation. [ 3.559154] ixgbe 0000:07:00.0: Adapter removed [ 3.559770] ixgbe: probe of 0000:07:00.0 failed with error -5 [ 3.575063] ixgbe 0000:07:00.1: Adapter removed [ 3.575511] ixgbe: probe of 0000:07:00.1 failed with error -5

But the card is there:

lspci | grep 10-Gigabit 07:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) 07:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

It might be worthwhile to mention that the two nodes are totally different machines (but both run PVE 7.0.1).

I know there are lots of X540-T2 fakes out there but the cards are genuine, as far as I can tell (they have yottamark stickers).

Any help appreciated!

 

[mira]

Is the 2nd node using the latest BIOS version? Latest NIC firmware?
If it's an older server, did you install the microcode package? https://wiki.debian.org/Microcode

 

[proxwolfe]

Is the 2nd node using the latest BIOS version?

It was not. I updated and now it seems to work (the dmesg output for the card now looks like on the first node)

Thank you for your help!

 

[proxwolfe]

Another server (not a node, this time), the same issue.

Updated BIOS/firmware. Installed microcode (server hardware's not too old; so just for good measure).

No dice.

This time, the trick was to disable "above 4G decoding" in the BIOS. (But updating may still have helped.)

 

[Glitch01]

Spot on. Disabled "above 4G decoding" in the bios; Intel 540 10 Gbe card didn't disconnect after complete bootup.