[SOLVED] Problems with pve-kernel-5.4.55-1-pve and ixgbe for intel x550

[Bengt Nolin]

Hello,

I just tried out the latest PVE kernel on the enterprise repo and after reboot my intel x550-t2 card does not work. If I reboot on the previous pve-kernel-5.4.44-2-pve it works as it should so the NIC is not broken.

The kernel ships with the ixgbe module but it doesn't load, and even if you load it using modprobe it doesn't matter since the NIC does not even show up in lspci.

From a working host (on previous kernel):

29:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
29:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)

root@proxmox3:~# dmesg | grep ixgbe
[ 0.995699] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[ 0.995699] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 1.696255] ixgbe 0000:29:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
[ 1.794212] ixgbe 0000:29:00.0: 31.504 Gb/s available PCIe bandwidth (8 GT/s x4 link)
[ 1.906204] ixgbe 0000:29:00.0: MAC: 4, PHY: 0, PBA No: H86377-005
[ 1.906205] ixgbe 0000:29:00.0: a0:36:9f:f2:e7:d4
[ 2.064884] ixgbe 0000:29:00.0: Intel(R) 10 Gigabit Network Connection
[ 2.064904] libphy: ixgbe-mdio: probed
[ 2.760496] ixgbe 0000:29:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
[ 2.858072] ixgbe 0000:29:00.1: 31.504 Gb/s available PCIe bandwidth (8 GT/s x4 link)
[ 2.970216] ixgbe 0000:29:00.1: MAC: 4, PHY: 0, PBA No: H86377-005
[ 2.970218] ixgbe 0000:29:00.1: a0:36:9f:f2:e7:d5
[ 3.130075] ixgbe 0000:29:00.1: Intel(R) 10 Gigabit Network Connection
[ 3.130093] libphy: ixgbe-mdio: probed
[ 3.130867] ixgbe 0000:29:00.0 enp41s0f0: renamed from eth0
[ 3.156378] ixgbe 0000:29:00.1 enp41s0f1: renamed from eth1
[ 96.445116] ixgbe 0000:29:00.0: registered PHC device on enp41s0f0
[ 96.578742] ixgbe 0000:29:00.1 enp41s0f1: changing MTU from 1500 to 9000
[ 96.678243] ixgbe 0000:29:00.1: registered PHC device on enp41s0f1
[ 101.713452] ixgbe 0000:29:00.0 enp41s0f0: NIC Link is Up 10 Gbps, Flow Control: None
[ 106.529006] ixgbe 0000:29:00.1 enp41s0f1: NIC Link is Up 10 Gbps, Flow Control: None

But on the new kernel "dmesg" doesn't show anything regarding ixgbe, and lspci does not list the card at all.

Edit: I can provide more HW info after the weekend, or open a bug report.

 

[t.lamprecht]

Hi,

An Intel X550T (not T2) works OK here with ixgbe and the pve-kernel-5.4.55-1-pve kernel, while not the exact same model it at least indicates that there is not a general regression here.

Can you please provide additional details about the setup? Vendor, CPU, ...
Is the firmware up to date?

Also, do you have some (remote) console access to check out the broken boot? I'd be interesting if there's anything in dmesg...

 

[Bengt Nolin]

I had time to test on another machine I almost forgot was running x550, and it has different (Intel) HW and works as expected.

I'll get back to you after the weekend with some logs from the AMD machine.

 

[t.lamprecht]

OK. FWIW, the machine I checked here is also AMD based (AMD EPYC 7351).

 

[Bengt Nolin]

So I got time to take a look at this machine again and today after rebooting on the newer kernel the NIC shows up!

This is A) good, B) weird.

I cannot reproduce the problem, but I have the kernel log from the problematic boot and I have compared it to today's boot. What's interesting is that pretty early you can see that memory is not reserved for the device and in the summary shortly after there is a difference in reserved memory.

Bad: Memory: 65697360K/67052404K available (14339K kernel code, 2366K rwdata, 4980K rodata, 2696K init, 5040K bss, 1355044K reserved, 0K cma-reserved)
Good: Memory: 65707924K/67052404K available (14339K kernel code, 2366K rwdata, 4980K rodata, 2696K init, 5040K bss, 1344480K reserved, 0K cma-reserved)

I think I'll treat this as a one-off and just document it for future reference.

 

[t.lamprecht]

Not that the system really didn't see it, i.e., like a loose connection or so.

But yeah, for now I'd also treat it as a fluke.