7.0.14-7 kernel breaks PCI link training

[aoxiangtianji]

I have a SK hynix PE4010 (HFS960GD0TEG-6410A, 894G) installed on a PCIe splitter card on an ASRock B450M Pro4-F. The splitter card has no switch chip; it relies on the motherboard's PCIe bifurcation.

After updating from 7.0.2 to 7.0.14-8, PVE got stuck during boot because it couldn't detect my disk. The NVMe controller is simply not visible, and I couldn't confirm any error in dmesg (in that state I wasn't able to export the log).

After some testing, I found that 7.0.14-6 is the latest kernel I can run.

From here it's out of my depth, so I relied heavily on AI tooling for the debugging — please bear with me. I had the AI filter the PCI-related patches between 7.0.14-6 and 7.0.14-7 to find which one caused this. It traced it to commit 72780f796468, and the reasoning is: the commit removed the ASM2824 whitelist and the link-up check, so the kernel now unconditionally lifts the firmware's 2.5 GT/s speed clamp and retrains the port to 8 GT/s during enumeration. On this link, Gen3 equalization never completes, so 8 GT/s is unreliable and the NVMe disappears.

To verify, I rebuilt 7.0.14-8 with a revert patch for that commit, reinstalled it, and it works now.

Attachments: pvereport, full lspci -vvv, dmesg of the reverted kernel, and the revert patch.

 

[fabian]

https://bugzilla.kernel.org/show_bug.cgi?id=221801
https://lore.kernel.org/20260801201244.4421-1-andiwild@gmail.com

could you maybe also try with the proposed fix and report whether it works to the kernel devs?

 

[svc]

I may have the same issue. With the 7.0.14-8 kernel, I am unable to boot a couple Supermicro SYS-E300-12D-10CN6P machines using bifurcation with AOC-SLG3-2M2 splitter cards (no active switch chips) and NVMe SSDs. I have 7.0.14-6 pinned until I can determine how to resolve.

 

[bindi]

FWIW 7.0.14-8-pve works and boots just fine for me. Asrock WRX80 Creator R2.0 and 4x1TB Corsair MP600 Pro LPX NVME on an Asus Hyper M.2 Gen4 card (x4x4x4x4, passive card)
E: also works with the latest 7.0.14-11-pve
E2: I am using systemd-boot, not GRUB

 

[svc]

A quick update to my inability to boot kernel 7.0.14-8 (above). The update to libpve-storage-perl to v.9.1.8 has resolved the problem and I can now boot kernel 7.0.14-8.

 

[fiona]

Hi @svc,
it's very unlikely that the libpve-storage-perl update is related. It's a perl library for high-level interaction with storages, it does not touch low-level kernel stuff related to booting. The changelog also doesn't mention anything like this: apt changelog libpve-storage-perl

 

[lama2176]

I seem to have a potentially related regression on a different platform, with the same kernel boundary.

Hardware:

• QDNV01 appliance/mainboard
• Intel Atom C3558 (Denverton), 4 cores
• AMI BIOS 5.13, dated 2024-02-21
• UEFI boot
• Proxmox VE 9.2
• pve-manager 9.2.10

Kernel results:

• 6.17.13-21-pve: boots successfully
• 7.0.14-6-pve: boots successfully
• 7.0.14-7-pve: fails to boot
• 7.0.14-8-pve: fails to boot
• 7.0.14-11-pve: fails to boot

The failure occurs extremely early. The console remains at the last GRUB message:

Loading initial ramdisk ...

There is no subsequent kernel output, and the host does not become reachable by ping or SSH.

Tests already performed:

• initramfs verified successfully with lsinitramfs
• all tested signed kernel packages pass dpkg -V
• sufficient free space in /boot and the EFI partition
• no DKMS modules
• nomodeset, loglevel=7, ignore_loglevel, initcall_debug, and keep_bootcon did not produce any additional kernel output
• dis_ucode_ldr did not change the behavior
• booting the failing kernel without initrd did not change the behavior
• blacklisting igc did not help
• acpi=off is not useful on this platform because even the otherwise working 7.0.14-6 kernel does not boot with it

The storage topology may make this relevant to this thread.

The PVE system/root disk is:

• Phison PS5013-E13 NVMe controller [1987:5013]
• PCI address 02:00.0
• directly attached to Intel Atom C3000 PCI Express Root Port #0 [8086:19a4] at 00:09.0
• no PCIe bifurcation card or external PCIe switch is involved
• the working kernel reports an 8 GT/s x2 link

There is also a second NVMe device:

• Kingston FURY Renegade / Phison E18 [2646:501f]
• 03:00.0, directly behind C3000 Root Port #1 at 00:0a.0
• also operating at 8 GT/s x2 with the working kernel

This appears to be a hardware-specific regression: the newer 7.0.14 kernels boot successfully on my other Proxmox nodes, although those use completely different hardware.
The exact regression boundary on this system is therefore:
7.0.14-6-pve = GOOD
7.0.14-7-pve = BAD
Given the identical regression boundary and the directly attached NVMe devices, could this be another manifestation of the PCIe link retraining regression discussed here?
I can provide lspci -vv, dmesg from the working 7.0.14-6 kernel, full DMI information, and the Proxmox version/package information. I can also test a patched/test kernel if that would help.

 

[svc]

Hi Fiona,

You are correct, the libpve-storage-perl update was not my issue.

I had also upgraded 4-year old mobo BIOS on both systems. I thought I had tested after the BIOS upgrade with the same boot failure, but obviously not. I downgraded libpve-storage-perl back to 9.1.7 and the system booted with 7.0.14-8-pve. I checked my apt log and libpve-storage-perl had been the only update on both systems, so the old BIOS must have been the issue.