7.0.14-7 kernel breaks NVMe link training

aoxiangtianji

Member
Aug 31, 2021
1
2
23
32
I have a SK hynix PE4010 (HFS960GD0TEG-6410A, 894G) installed on a PCIe splitter card on an ASRock B450M Pro4-F. The splitter card has no switch chip; it relies on the motherboard's PCIe bifurcation.

After updating from 7.0.2 to 7.0.14-8, PVE got stuck during boot because it couldn't detect my disk. The NVMe controller is simply not visible, and I couldn't confirm any error in dmesg (in that state I wasn't able to export the log).

After some testing, I found that 7.0.14-6 is the latest kernel I can run.

From here it's out of my depth, so I relied heavily on AI tooling for the debugging — please bear with me. I had the AI filter the PCI-related patches between 7.0.14-6 and 7.0.14-7 to find which one caused this. It traced it to commit 72780f796468, and the reasoning is: the commit removed the ASM2824 whitelist and the link-up check, so the kernel now unconditionally lifts the firmware's 2.5 GT/s speed clamp and retrains the port to 8 GT/s during enumeration. On this link, Gen3 equalization never completes, so 8 GT/s is unreliable and the NVMe disappears.

To verify, I rebuilt 7.0.14-8 with a revert patch for that commit, reinstalled it, and it works now.

Attachments: pvereport, full lspci -vvv, dmesg of the reverted kernel, and the revert patch.
 

Attachments

I may have the same issue. With the 7.0.14-8 kernel, I am unable to boot a couple Supermicro SYS-E300-12D-10CN6P machines using bifurcation with AOC-SLG3-2M2 splitter cards (no active switch chips) and NVMe SSDs. I have 7.0.14-6 pinned until I can determine how to resolve.
 
FWIW 7.0.14-8-pve works and boots just fine for me. Asrock WRX80 Creator R2.0 and 4x1TB Corsair MP600 Pro LPX NVME on an Asus Hyper M.2 Gen4 card (x4x4x4x4, passive card)