7.0.14-7 kernel breaks PCI link training

A

aoxiangtianji

Member
Aug 31, 2021
1
3
23
32
I have a SK hynix PE4010 (HFS960GD0TEG-6410A, 894G) installed on a PCIe splitter card on an ASRock B450M Pro4-F. The splitter card has no switch chip; it relies on the motherboard's PCIe bifurcation.

After updating from 7.0.2 to 7.0.14-8, PVE got stuck during boot because it couldn't detect my disk. The NVMe controller is simply not visible, and I couldn't confirm any error in dmesg (in that state I wasn't able to export the log).

After some testing, I found that 7.0.14-6 is the latest kernel I can run.

From here it's out of my depth, so I relied heavily on AI tooling for the debugging — please bear with me. I had the AI filter the PCI-related patches between 7.0.14-6 and 7.0.14-7 to find which one caused this. It traced it to commit 72780f796468, and the reasoning is: the commit removed the ASM2824 whitelist and the link-up check, so the kernel now unconditionally lifts the firmware's 2.5 GT/s speed clamp and retrains the port to 8 GT/s during enumeration. On this link, Gen3 equalization never completes, so 8 GT/s is unreliable and the NVMe disappears.

To verify, I rebuilt 7.0.14-8 with a revert patch for that commit, reinstalled it, and it works now.

Attachments: pvereport, full lspci -vvv, dmesg of the reverted kernel, and the revert patch.
 

Attachments

  • Like
Reactions: lama2176, MusikJunk and Onslow
I may have the same issue. With the 7.0.14-8 kernel, I am unable to boot a couple Supermicro SYS-E300-12D-10CN6P machines using bifurcation with AOC-SLG3-2M2 splitter cards (no active switch chips) and NVMe SSDs. I have 7.0.14-6 pinned until I can determine how to resolve.
 
FWIW 7.0.14-8-pve works and boots just fine for me. Asrock WRX80 Creator R2.0 and 4x1TB Corsair MP600 Pro LPX NVME on an Asus Hyper M.2 Gen4 card (x4x4x4x4, passive card)
E: also works with the latest 7.0.14-11-pve
E2: I am using systemd-boot, not GRUB
 
Last edited:
A quick update to my inability to boot kernel 7.0.14-8 (above). The update to libpve-storage-perl to v.9.1.8 has resolved the problem and I can now boot kernel 7.0.14-8.
 
Hi @svc,
it's very unlikely that the libpve-storage-perl update is related. It's a perl library for high-level interaction with storages, it does not touch low-level kernel stuff related to booting. The changelog also doesn't mention anything like this: apt changelog libpve-storage-perl
 
I seem to have a potentially related regression on a different platform, with the same kernel boundary.

Hardware:
  • QDNV01 appliance/mainboard
  • Intel Atom C3558 (Denverton), 4 cores
  • AMI BIOS 5.13, dated 2024-02-21
  • UEFI boot
  • Proxmox VE 9.2
  • pve-manager 9.2.10
Kernel results:
  • 6.17.13-21-pve: boots successfully
  • 7.0.14-6-pve: boots successfully
  • 7.0.14-7-pve: fails to boot
  • 7.0.14-8-pve: fails to boot
  • 7.0.14-11-pve: fails to boot
The failure occurs extremely early. The console remains at the last GRUB message:
Loading initial ramdisk ...
Click to expand...
There is no subsequent kernel output, and the host does not become reachable by ping or SSH.

Tests already performed:
  • initramfs verified successfully with lsinitramfs
  • all tested signed kernel packages pass dpkg -V
  • sufficient free space in /boot and the EFI partition
  • no DKMS modules
  • nomodeset, loglevel=7, ignore_loglevel, initcall_debug, and keep_bootcon did not produce any additional kernel output
  • dis_ucode_ldr did not change the behavior
  • booting the failing kernel without initrd did not change the behavior
  • blacklisting igc did not help
  • acpi=off is not useful on this platform because even the otherwise working 7.0.14-6 kernel does not boot with it
The storage topology may make this relevant to this thread.

The PVE system/root disk is:
  • Phison PS5013-E13 NVMe controller [1987:5013]
  • PCI address 02:00.0
  • directly attached to Intel Atom C3000 PCI Express Root Port #0 [8086:19a4] at 00:09.0
  • no PCIe bifurcation card or external PCIe switch is involved
  • the working kernel reports an 8 GT/s x2 link
There is also a second NVMe device:
  • Kingston FURY Renegade / Phison E18 [2646:501f]
  • 03:00.0, directly behind C3000 Root Port #1 at 00:0a.0
  • also operating at 8 GT/s x2 with the working kernel
This appears to be a hardware-specific regression: the newer 7.0.14 kernels boot successfully on my other Proxmox nodes, although those use completely different hardware.
The exact regression boundary on this system is therefore:
7.0.14-6-pve = GOOD
7.0.14-7-pve = BAD
Given the identical regression boundary and the directly attached NVMe devices, could this be another manifestation of the PCIe link retraining regression discussed here?
I can provide lspci -vv, dmesg from the working 7.0.14-6 kernel, full DMI information, and the Proxmox version/package information. I can also test a patched/test kernel if that would help.
 
  • Like
Reactions: MusikJunk
Hi Fiona,

You are correct, the libpve-storage-perl update was not my issue.

I had also upgraded 4-year old mobo BIOS on both systems. I thought I had tested after the BIOS upgrade with the same boot failure, but obviously not. I downgraded libpve-storage-perl back to 9.1.7 and the system booted with 7.0.14-8-pve. I checked my apt log and libpve-storage-perl had been the only update on both systems, so the old BIOS must have been the issue.
 
  • Like
Reactions: fiona
lama2176 said:
I seem to have a potentially related regression on a different platform, with the same kernel boundary.

Hardware:
  • QDNV01 appliance/mainboard
  • Intel Atom C3558 (Denverton), 4 cores
  • AMI BIOS 5.13, dated 2024-02-21
  • UEFI boot
Click to expand...

I can duplicate that on the same motherboard - the only difference being a C3758R CPU (8 Cores/2.4ghz)

The machine hangs when booting 7.0.14-11-pve, with no further output after `Loading initial ramdisk ...`

It *does* boot when using 7.0.14-6-pve.

Annoyingly, I can't seem to get the kernel param 'earlyprintk' to even work with either kernels, but I'm still investigating.
 
Last edited:
  • Like
Reactions: lama2176
lama2176 said:
I seem to have a potentially related regression on a different platform, with the same kernel boundary.

Hardware:
  • QDNV01 appliance/mainboard
  • Intel Atom C3558 (Denverton), 4 cores
  • AMI BIOS 5.13, dated 2024-02-21
  • UEFI boot
  • Proxmox VE 9.2
  • pve-manager 9.2.10
Kernel results:
  • 6.17.13-21-pve: boots successfully
  • 7.0.14-6-pve: boots successfully
  • 7.0.14-7-pve: fails to boot
  • 7.0.14-8-pve: fails to boot
  • 7.0.14-11-pve: fails to boot
The failure occurs extremely early. The console remains at the last GRUB message:

There is no subsequent kernel output, and the host does not become reachable by ping or SSH.

Tests already performed:
  • initramfs verified successfully with lsinitramfs
  • all tested signed kernel packages pass dpkg -V
  • sufficient free space in /boot and the EFI partition
  • no DKMS modules
  • nomodeset, loglevel=7, ignore_loglevel, initcall_debug, and keep_bootcon did not produce any additional kernel output
  • dis_ucode_ldr did not change the behavior
  • booting the failing kernel without initrd did not change the behavior
  • blacklisting igc did not help
  • acpi=off is not useful on this platform because even the otherwise working 7.0.14-6 kernel does not boot with it
The storage topology may make this relevant to this thread.

The PVE system/root disk is:
  • Phison PS5013-E13 NVMe controller [1987:5013]
  • PCI address 02:00.0
  • directly attached to Intel Atom C3000 PCI Express Root Port #0 [8086:19a4] at 00:09.0
  • no PCIe bifurcation card or external PCIe switch is involved
  • the working kernel reports an 8 GT/s x2 link
There is also a second NVMe device:
  • Kingston FURY Renegade / Phison E18 [2646:501f]
  • 03:00.0, directly behind C3000 Root Port #1 at 00:0a.0
  • also operating at 8 GT/s x2 with the working kernel
This appears to be a hardware-specific regression: the newer 7.0.14 kernels boot successfully on my other Proxmox nodes, although those use completely different hardware.
The exact regression boundary on this system is therefore:
7.0.14-6-pve = GOOD
7.0.14-7-pve = BAD
Given the identical regression boundary and the directly attached NVMe devices, could this be another manifestation of the PCIe link retraining regression discussed here?
I can provide lspci -vv, dmesg from the working 7.0.14-6 kernel, full DMI information, and the Proxmox version/package information. I can also test a patched/test kernel if that would help.
Click to expand...

Qotom support confirmed a boot-problem with 7-0-14-8, but they found another problem with serial, see:

https://www.qotom.com/support/kerne...n-proxmox-ve-9-2-with-pve-kernel-7-0-14-8-pve
 
  • Like
Reactions: lama2176 and xrobau
MusikJunk said:
Qotom support confirmed a boot-problem with 7-0-14-8, but they found another problem with serial, see:

https://www.qotom.com/support/kerne...n-proxmox-ve-9-2-with-pve-kernel-7-0-14-8-pve
Click to expand...
Confirmed fixed for my specific issue - NOT the original post, unfortunately.

Adding that to /etc/default/grub and running
Code: 
proxmox-boot-tool refresh
lets it boot with 7.0.14-11-pve

Here's mine:
Code: 
GRUB_CMDLINE_LINUX="mitigations=off init_on_alloc=0 nomodeset pcie_aspm=off initcall_blacklist=mid8250_pci_driver_init"

(Note you only want 'mitigations=off' if you 100% trust everyone with access to vms running on the machine)
 
  • Like
Reactions: lama2176 and MusikJunk
Solved for me using the Qotom workaround.

On my QDNV01 / Intel Atom C3558 system, adding:

initcall_blacklist=mid8250_pci_driver_init
Click to expand...
allows 7.0.14-11-pve to boot normally.

Confirmed in dmesg:
blacklisting initcall mid8250_pci_driver_init
initcall mid8250_pci_driver_init blacklisted
Click to expand...
The affected device is the Intel C3000 HSUART controller (8086:19d8), normally handled by 8250_mid and exposed as ttyS4.

I do not use this serial port, so the workaround has no practical impact for me. Host, networking, storage and VMs/CTs are running normally.
 
  • Like
Reactions: MusikJunk
Hi,
the patch is included in
Code: 
proxmox-kernel-7.0 (7.0.14-12) trixie; urgency=medium

  * fix #7905: cherry-pick fix for null-ptr deref in amdgpu

  * fix #7872: cherry-pick fix for intel denverton serial driver.

  * cherry-pick fixes for CVEs announced since 08.08.2026

 -- Proxmox Support Team <support@proxmox.com>  Tue, 11 Aug 2026 13:05:38 +0200
currently available in the pve-test repository.
 
fiona said:
Hi,
the patch is included in
Code: 
proxmox-kernel-7.0 (7.0.14-12) trixie; urgency=medium

  * fix #7905: cherry-pick fix for null-ptr deref in amdgpu

  * fix #7872: cherry-pick fix for intel denverton serial driver.

  * cherry-pick fixes for CVEs announced since 08.08.2026

 -- Proxmox Support Team <support@proxmox.com>  Tue, 11 Aug 2026 13:05:38 +0200
currently available in the pve-test repository.
Click to expand...

You can mark that ticket as 'fixed'. Thanks!

Code: 
root@smallbox:~# dmesg | grep TI167
[    0.864184] 0000:00:1a.0: ttyS4 at MMIO 0xdf619000 (irq = 16, base_baud = 115200) is a TI16750
root@smallbox:~# uname -a
Linux smallbox 7.0.14-12-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-12 (2026-08-11T11:05Z) x86_64 GNU/Linux
root@smallbox:~# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-7.0.14-12-pve root=/dev/mapper/smallbox--vg-root ro mitigations=off init_on_alloc=0 nomodeset pcie_aspm=off
root@smallbox:~# grep model.name /proc/cpuinfo | head -1
model name      : Intel(R) Atom(TM) CPU C3758R @ 2.40GHz
root@smallbox:~#
 
You must log in or register to reply here.
Top Bottom
Back