PVE 9.2/Kernel 7.0.14-26: Kingston DC2000B NVMe initialization fails for one of two drives

[MarkDotDK]

I'm in the progress of setting up a new home lab host for Proxmox, but I am hitting a couple of issues, that I think is related to a bug/regression in the NVMe kernel driver.
The setup in short:

Motherboard: Supermicro X14SBI-F
CPU: Xeon 6511P
Mem: 128GB Micron
Boot drives: 2x Kingston DC2000B M.2 240GB drives attached directly in the M.2 slots on the motherboard

In summary, I am facing issues getting the kernel to tecognize both NVMe drives under current 7.0.14-26 kernel, the the first drive fail initialization, but the other works just fine. The symptoms change based on the kernel. When I originally booted on the 9.2 installer with the default 7.0.2-6 kernel, neither of the drives were detected at all, so couldn't even install Proxmox. Going back a step to 9.1 installer, which uses 6.x kernel, all works well. Both drives are detected, and I was successfully able to install Proxmox in a ZFS RAID1 setup on the two drives. I then immediately did upgrade to 9.2. At the next boot, only one of the NVMe drives are enabled, so the ZFS pool is degraded. During boot, the machine appears to be hanging for around 60 seconds during kernel initialization (as it probes the hardware) before continuing. I believe this is a standard NVMe timeout of some sort?

This is representative for what is shown in the boot logs every time:

[ 75.434319] pm-core-2 kernel: nvme nvme0: pci function 0000:2c:00.0
[ 75.434434] pm-core-2 kernel: nvme nvme1: pci function 0000:2d:00.0
[ 75.434646] pm-core-2 kernel: nvme nvme1: D3 entry latency set to 10 seconds
[ 75.435229] pm-core-2 kernel: nvme nvme1: 32/0/0 default/read/poll queues
[ 75.435234] pm-core-2 kernel: nvme1n1: p2 p3
[ 75.438281] pm-core-2 kernel: nvme nvme0: missing or invalid SUBNQN field.
[ 75.440017] pm-core-2 kernel: nvme nvme0: request 0x0 genctr mismatch (got 0x0 expected 0x1)
[ 75.440032] pm-core-2 kernel: nvme nvme0: invalid id 0 completed on queue 0
[ 75.441077] pm-core-2 kernel: nvme nvme0: I/O tag 1 (0001) QID 0 timeout, disable controller
[ 75.441093] pm-core-2 kernel: nvme nvme0: Failed to read smart log (error -4)
[ 75.441162] pm-core-2 kernel: nvme 0000:2c:00.0: error -EINTR: probe failed
[ 75.441231] pm-core-2 kernel: nvme 0000:2c:00.0: probe with driver nvme failed with error -4

I have rebooted the machine more times than I can count, and it has always been the device at 0000:2c:00.0 that doesn't come online. 0000:2d:00.0 is as happy as can be.

I have poked around BIOS settings to see if there was anhything that could fix it, but no luck. I have also tried to slow down the PCIe links to 8 GT/s to rule out signal integrity issues, but no change there either. It is expected that the drives only report a x2 link, as Supermicro has only wired two lanes to each of the M.2 slots.

The "good" news is that there is a workaround to get both drives online under 7.0.14-26. If I reset the controller and re-bind it like this:

echo 1 > /sys/bus/pci/devices/0000:2c:00.0/reset
echo 0000:2c:00.0 > /sys/bus/pci/drivers/nvme/bind

This brings the controller and drive online instantly, ZFS sees the disk, resilvers it, and all is good. Until next reboot of course.

Doing a PCI device removal and rescan correctly discovers the controller, but initialization still fails (and there's again this 60-second wait):

echo 1 > /sys/bus/pci/devices/0000:2c:00.0/remove
echo 1 > /sys/bus/pci/rescan

This gets accompanied by the same log entries as above:

missing or invalid SUBNQN field
request 0x0 genctr mismatch (got 0x0 expected 0x1)
invalid id 0 completed on queue 0

Nothing is running on this machine yet, so I am fine doing experiments on it if that can help narrow down the issue. Just let me know what could be needed.

The ultimate fix is pinning kernel 6.17.13-23 - everything works here. There's no 60-second delay during boot, and both drives are good and ZFS all happy without. Apparently something changed around NVMe controller initialization, reset sequencing, and/or PCIe/DMA handling between the 6.17 kernel and the 7.0 kernel versions. Considering that 7.0.2-6 fails to initialize both controllers, there has been some improvements .

I have attached lspci output from a broken state (0000:2c:00.0 offline, 0000:2d:00.0 working) along with boot journals from a kernel 6.17.2-1, kernel 6.17.13-21 and the current 7.0.14-26. In the log for 7.0.14-26 you can see that I perform the reset/rebinding around second 3975.

Not sure if this is even relevant, but I also noted that smartctl reports command failures:
Error Information (NVMe Log 0x01, 16 of 63 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS Message
0 56 0 0x0000 0x4004 0x028 0 0 - Invalid Field in Command
1 55 0 0x000b 0x4004 - 0 0 - Invalid Field in Command
Error Information (NVMe Log 0x01, 16 of 63 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS Message
0 68 0 0x000c 0x4004 0x028 0 0 - Invalid Field in Command
1 67 0 0x001b 0x4004 - 0 0 - Invalid Field in Command

I wonder if this is related to the recent changes/fixes to PCIe link training that affected some of the recent kernels?

Thanks for listening to my "rant"...

/Mark

 

[fabian]

could you try an upstream mainline kernel (warning - no ZFS!)? if it also fails there, please report it upstream... it's possible this device requires a quirk on the kernel side.

 

[MarkDotDK]

Thanks for the suggestion. I spend quite a lot of time yesterday and today booting various mainline/generic kernels (7.0.14, 7.1.13, 7.2.15, 7.3.1, 7.4.0rc1) and trying enabling the various NVMe quirks. Unfortunately, none of them worked, so I started digging into the kernel sources to see what relevant commits have been made since 6.17.13 and 7.0.14 (and never), and I found a specific commit (a22250fe933d) that implements a quirk specifically targeting Xeon 6 platforms and PCIe devices with x2 link speeds. The patch is specifically designed to improve performance by limiting the MRRS and disabling Extended tags on the PCIe side of things.

Since I am on a Xeon 6 with the two Kingston drives trained up with exactly a x2 link (Supermicro only wired the two M.2 slots for x2 as they are likely inteded as boot drives and don't need stellaer performance - the two slots share the same PCIe root port along with a SATA controller) I decided to look more into this patch. I then found a subsequent patch proposal from August 2026 (https://lore.kernel.org/linux-pci/20260804204845.171483-1-rick@microway.com/) that addresses the first patch's very broad scope and tries to narrow it down a bit.

I then checked out the PVE kernel source tree for 7.0.14-26 in two distinct trees, on tree A, I removed commit a22250fe933d completely from arch/x86/pci/fixup.c, and in tree B, I applied the proposed v2 patch and compiled both kernels. I then installed them on the PVE host and tried booting each of these kernels.

For tree B with v2 patch applied, the issue remains. NVMe controllers fail to initialize unless they are are sent a PCI reset and then bound to the NVMe driver. But on tree A, both drives initialize cleanly without any issues at all, so this quirk is the root cause of the issues I am seeing. This is not an NVMe driver issue, as I first assumed. It is a PCIe initialization issue, which obviously changes things a bit.

I know this isn't a Proxmox issue at all, but I still figured I'd give an update here in case someone else stumbles into similar issues. If you are running on Xeon 6 with an SSD running at x2 speeds running any 7.x kernel (the patch is not in 6.7.13), you are likely to be affected. I will try to figure out how to report this upstream.

An interesting aspect is of course that PVE kernel 7.0.14-26 boots with one of the drives successfully initialized, allowing the machine to boot . I can then do the FLR reset on the drive that is missing and get it to work. Something in the Proxmox-applied patches must impact the PCIe initialization procedurein some way.

/Mark

 

[cwt]

Short question: did you set in the BIOS under Advanced -> IIO Configuration this

Equalization Bypass To Highest Rate -> Enabled

If not, try to enable it. Another option is PCIe completion timeout.

 

[MarkDotDK]

Short question: did you set in the BIOS under Advanced -> IIO Configuration this

Equalization Bypass To Highest Rate -> Enabled

If not, try to enable it. Another option is PCIe completion timeout.

I did try having Equalization Bypass To Highest Rate set to both Enabled and Disable without luck. PCIe completion timeout is set to 250 to 990 (or something like that - it's the default value). The kernel PCI subsystem mailing list does acknowledge that there are some collateral damage because of the Xeon 6-related quirk. I am tagging on to that message thread. The important detail is that the M.2 slots are physically wired for 2x only - and the Xeon 6 quirk is targeting cases where things run x2 like my M.2 drives are. Right now the theory is that as the patch is disabling extended tags, the NVMe controller fails to initialize. The patch author specifically said he had encountered that, so right now things are pointing very much in that direction.

I will keep this thread post updated with any progress on the upstream kernel side.