Proxmox 8.3 - Intel X710 Network Adapter not working

[dusty128]

Dear Proxmox Forum,
I am trying to setup a new cluster consisting of 3 Supermicro X13-SAE-F Mainboards. I also installed a common Intel X710 Network adapter for 10G networking (for Ceph):

02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev
02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-2 [8086:0008]
Kernel modules: i40e
02:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev
02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710 [8086:0000]
Kernel modules: i40e

When booting the Linux kernel (Debian / Proxmox) the system runs into some timeouts and networking is down. The reason lies in the loading of the kernel module for the Intel X710 network adapter (i40e). When loading this module it tells in the log:

[Sun Dec 29 23:54:15 2024] i40e 0000:02:00.0: Unable to change power state from D3cold to D0, device inaccessible
[Sun Dec 29 23:56:26 2024] i40e 0000:02:00.0: PF reset failed: -5
[Sun Dec 29 23:56:26 2024] i40e 0000:02:00.0: Entering recovery mode due to repeated FW resets. This may take several minutes. Refer to the Intel(R) Ethernet Adapters and Devices User Guide.
[Sun Dec 29 23:56:26 2024] i40e: probe of 0000:02:00.0 failed with error -5
[Sun Dec 29 23:56:26 2024] i40e 0000:02:00.1: Unable to change power state from D3cold to D0, device inaccessible
[Sun Dec 29 23:58:37 2024] i40e 0000:02:00.1: PF reset failed: -5
[Sun Dec 29 23:58:37 2024] i40e 0000:02:00.1: Entering recovery mode due to repeated FW resets. This may take several minutes. Refer to the Intel(R) Ethernet Adapters and Devices User Guide.
[Sun Dec 29 23:58:37 2024] i40e: probe of 0000:02:00.1 failed with error -5

Our Linux kernel version is 6.8.12-4-pve

Some similar problems seem to have been fixed by adding the following as kernel boot parameter:

pcie_port_pm=off

Unfortunately in this case, it does not help.

Any clue how to solve this problem?
Regards,
Hermann

 

[dusty128]

O.k., I found a solution by myself and let it know for others:

I got feedback from Supermicro regarding this problem:

- Supermicro testet the board (X13-SAE-F) with the Intel X710 with Ubuntu 22.04 (kernel 5.15) only.
- Supermicro can confirm that the Intel X710 does not work with Ubuntu 6.8 and state, that there will be no support due to design limitations.

So basically this means that there is no support from Supermicro for these Network Adapters for Kernels > 5.15 and thus no support for Proxmox 8.

However, I found a solution: Initially we placed the network adapter in slot 4 (middle) which is not working. Now we tried it with slot 7 (rightmost):

Although this is untested by Supermicro, it now works.

-> So in case you have a similar problem, try to change the PCIe slot.

Best Regards,
Hermann

 

[lebowski89]

Hi Hermann,

Have an X710-DA4 in the pcie4x16 slot of a X570-P. Updated its firmware to the latest in another (Windows) machine before adding it. Was detected by Proxmox right away, and each port is in its own iommu group. Have successfully passed through a port to an UnRaid VM. But I have noticed these messages under the Proxmox host:






0a:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-4
Flags: fast devsel, IRQ 115, IOMMU group 25
Memory at fa800000 (64-bit, prefetchable) [size=8M]
Memory at fb818000 (64-bit, prefetchable) [size=32K]
Expansion ROM at fce80000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable- Count=129 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 98-12-d0-ff-ff-fe-fd-3c
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1b0] Access Control Services
Capabilities: [1d0] Secondary PCI Express
Kernel driver in use: vfio-pci
Kernel modules: i40e

0a:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710
Flags: fast devsel, IRQ 115, IOMMU group 26
Memory at fa000000 (64-bit, prefetchable) [size=8M]
Memory at fb810000 (64-bit, prefetchable) [size=32K]
Expansion ROM at fce00000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable- Count=129 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 98-12-d0-ff-ff-fe-fd-3c
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1b0] Access Control Services
Kernel driver in use: vfio-pci
Kernel modules: i40e

0a:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710
Flags: bus master, fast devsel, latency 0, IRQ 115, IOMMU group 27
Memory at f9800000 (64-bit, prefetchable) [size=8M]
Memory at fb808000 (64-bit, prefetchable) [size=32K]
Expansion ROM at fcd80000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 98-12-d0-ff-ff-fe-fd-3c
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1b0] Access Control Services
Kernel driver in use: vfio-pci
Kernel modules: i40e

0a:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710
Flags: fast devsel, IRQ 115, IOMMU group 28
Memory at f9000000 (64-bit, prefetchable) [size=8M]
Memory at fb800000 (64-bit, prefetchable) [size=32K]
Expansion ROM at fcd00000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable- Count=129 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 98-12-d0-ff-ff-fe-fd-3c
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1b0] Access Control Services
Kernel driver in use: vfio-pci
Kernel modules: i40e

I wonder if its just an X710 issue? The card itself seems to be working so far, despite the above logs - but I haven't tested much.

 

[dusty128]

Thank you for your reply, this is interesting!

I use on my X13-SAE-F BIOS-Version 4.1. The newest one available at Supermicro is 4.3 - eventually something was fixed? I won't test that, however, as everything runs fine now with the X-710 in the other slot as depicted.

Regarding your error messages I wonder that AMD-Vi logs are shown. AFAIK AMD-Vi is the I/O virtualization tech by AMD, whereas Intel has VT-d? Why is AMD technology displayed on an Intel mainboard?

Anyway - I wonder if that has something to do with IOMMU. Maybe it is an option to switch this off in the BIOS.

Best Regards,
Hermann

 

[lebowski89]

Why is AMD technology displayed on an Intel mainboard?

It is an AMD mainboard. X570-P is AM4. CPU is 5900XT. Yeah, after testing, I worked out that Debian 12 doesn't really play nice with X710 - it detected the passed through port, but the interface was listed as unclaimed within the VM. Switching to a Ubuntu 24.04 LTS VM, the X710 was detected correctly, and successfully was listed as a network interface. Ever since passing through to the Ubuntu VM, those error messages have stopped. So i guess it was the VM.

(Replied to this topic because you use the X710, not because you use the Supermicro, sorry if that caused any confusion)