[SOLVED] Dell R710 & Intel X710

[chrigiboy]

Hello,

Recently we bought Intel X710 Network Cards to upgrade our cluster to 10GBe.
On about half of all our Servers the new Network Card got detected and is showing up in Proxmox.
The Servers on which the network card isn't showing up are all Dell R710 Servers.
I have checked wether there are differences in software versions but on all systems (working and not working) there's the same version installed:
PVE: pve-manager/6.4-8/185e14db
Kernel: 5.4.114-1-pve
I40e Mod: 2.8.20-k

The I40e Mod is installed on all systems (working and not working):

$ modinfo i40e
filename:       /lib/modules/5.4.114-1-pve/kernel/drivers/net/ethernet/intel/i40e/i40e.ko
version:        2.8.20-k
license:        GPL v2
description:    Intel(R) Ethernet Connection XL710 Network Driver
author:         Intel Corporation, <e1000-devel@lists.sourceforge.net>
srcversion:     75DF954553B71FF0423AAFA
alias:          pci:v00008086d0000158Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000158Asv*sd*bc*sc*i*
alias:          pci:v00008086d00000D58sv*sd*bc*sc*i*
alias:          pci:v00008086d00000CF8sv*sd*bc*sc*i*
alias:          pci:v00008086d00001588sv*sd*bc*sc*i*
alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D3sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D2sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D1sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D0sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CFsv*sd*bc*sc*i*
alias:          pci:v00008086d000037CEsv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Esv*sd*bc*sc*i*
alias:          pci:v00008086d000015FFsv*sd*bc*sc*i*
alias:          pci:v00008086d00001589sv*sd*bc*sc*i*
alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
depends:       
retpoline:      Y
intree:         Y
name:           i40e
vermagic:       5.4.114-1-pve SMP mod_unload modversions
parm:           debug:Debug level (0=none,...,16=all), Debug mask (0x8XXXXXXX) (uint)

The new network card hasn't added any network interfaces.
All commands (ifconfig, net link show) just don't output anything coming from the new Network Card (X710).
With the command "lshw" it's showing "UNCLAIMED" for these two interfaces.

 $ lshw -class network
  *-network:0 UNCLAIMED
       description: Ethernet controller
       product: Ethernet Controller X710 for 10GbE SFP+
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:09:00.0
       version: 02
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd cap_list
       configuration: latency=0
       resources: memory:ef100000-ef17ffff
  *-network:1 UNCLAIMED
       description: Ethernet controller
       product: Ethernet Controller X710 for 10GbE SFP+
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:09:00.1
       version: 02
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd cap_list
       configuration: latency=0
       resources: memory:ef180000-ef1fffff

So basically it's working on all of our Server which aren't a Dell R710 Server and running Proxmox.
We have one other dell R710 Server which is running Windows on which the Network card is also working as expected.
Why is lshw displaying "UNCLAIMED" even though mod i40e is installed?
What can we do?

 

[aaron]

AFAICT you are using the 5.4 kernel. You could try to see if the newer 5.11 kernel works: apt install pve-kernel-5.11 and reboot the nodes.

What are the other nodes where everything seems to work?

 

[chrigiboy]

Hey,

Thanks for your reply.
I've checked all our proxmox nodes.
They're all running on the exact same Kernel Version "5.4.114-1-pve" (working ones and servers which can't see the interfaces).
I'll perform some tests with other Kernel versions but as we have machines which do work with this exact Kernel Version i don't think this will solve our problem. Any other ideas?

 

[argonius]

Hey chrigiboy,
maybe stupid question, but have you plugged the card from the working r710 (windows) to one of the proxmox r710?
do you use same sfp's in the cards?
greetz

 

[chrigiboy]

Hello argonius.
It IS the same server. I booted with a Windows on a USB Stick, and the network device was working without any issues.

 

[argonius]

uh. weird.
have you tried latest version of i40e?
https://downloadcenter.intel.com/do...bit-Ethernet-Network-Connections-Under-Linux-

 

[chrigiboy]

Yes, self compiled with the same issue.

 

[argonius]

whats the output for

lspci -v -s 09:00.0

and is it visible in dmesg?

 

[chrigiboy]

:~# lspci -v -s 09:00.0
09:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-2
Flags: fast devsel, IRQ 48, NUMA node 0, IOMMU group 18
Memory at <ignored> (64-bit, prefetchable)
Memory at <ignored> (64-bit, prefetchable)
Expansion ROM at ec900000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable- Count=129 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-6c-3c-ff-ff-b7-aa-40
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1b0] Access Control Services
Capabilities: [1d0] Secondary PCI Express
Kernel modules: i40e

 

[argonius]

does

lspci -nn | grep X710

provides you the correct device id?
have you tried to search for the deviceID if there is maybe any issue from identifying this deviceID for the i40e module?

regarding your latest post: i hoped the kernel didnt just loaded the i40e module as it is not finding the correct module for this device :/

 

[chrigiboy]

Yes:
~# lspci -nn | grep X710
09:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 02)
09:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 02)

 

[argonius]

tbh no clue why it is "unclaimed" but lspci is showing it is using i40e driver...

 

[chrigiboy]

It looks like, its a Proxmox and Intel Bug. We have changed to x520 DA2 (PCIE v2.0), which will work.

 

[argonius]

wow, thanks for the update.
have you also booted "plain" debian, so you can say it is proxmox or linux issue?

 

[chrigiboy]

Oh i didnt test it with debian. So i am not sure if it would work with debian or not.