Proxmox 8.3 - Intel X710 Network Adapter not working

[dusty128]

Dear Proxmox Forum,
I am trying to setup a new cluster consisting of 3 Supermicro X13-SAE-F Mainboards. I also installed a common Intel X710 Network adapter for 10G networking (for Ceph):

02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev
02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-2 [8086:0008]
Kernel modules: i40e
02:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev
02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710 [8086:0000]
Kernel modules: i40e

When booting the Linux kernel (Debian / Proxmox) the system runs into some timeouts and networking is down. The reason lies in the loading of the kernel module for the Intel X710 network adapter (i40e). When loading this module it tells in the log:

[Sun Dec 29 23:54:15 2024] i40e 0000:02:00.0: Unable to change power state from D3cold to D0, device inaccessible
[Sun Dec 29 23:56:26 2024] i40e 0000:02:00.0: PF reset failed: -5
[Sun Dec 29 23:56:26 2024] i40e 0000:02:00.0: Entering recovery mode due to repeated FW resets. This may take several minutes. Refer to the Intel(R) Ethernet Adapters and Devices User Guide.
[Sun Dec 29 23:56:26 2024] i40e: probe of 0000:02:00.0 failed with error -5
[Sun Dec 29 23:56:26 2024] i40e 0000:02:00.1: Unable to change power state from D3cold to D0, device inaccessible
[Sun Dec 29 23:58:37 2024] i40e 0000:02:00.1: PF reset failed: -5
[Sun Dec 29 23:58:37 2024] i40e 0000:02:00.1: Entering recovery mode due to repeated FW resets. This may take several minutes. Refer to the Intel(R) Ethernet Adapters and Devices User Guide.
[Sun Dec 29 23:58:37 2024] i40e: probe of 0000:02:00.1 failed with error -5

Our Linux kernel version is 6.8.12-4-pve

Some similar problems seem to have been fixed by adding the following as kernel boot parameter:

pcie_port_pm=off

Unfortunately in this case, it does not help.

Any clue how to solve this problem?
Regards,
Hermann

 

[dusty128]

O.k., I found a solution by myself and let it know for others:

I got feedback from Supermicro regarding this problem:

- Supermicro testet the board (X13-SAE-F) with the Intel X710 with Ubuntu 22.04 (kernel 5.15) only.
- Supermicro can confirm that the Intel X710 does not work with Ubuntu 6.8 and state, that there will be no support due to design limitations.

So basically this means that there is no support from Supermicro for these Network Adapters for Kernels > 5.15 and thus no support for Proxmox 8.

However, I found a solution: Initially we placed the network adapter in slot 4 (middle) which is not working. Now we tried it with slot 7 (rightmost):

Although this is untested by Supermicro, it now works.

-> So in case you have a similar problem, try to change the PCIe slot.

Best Regards,
Hermann