Kernel 7.0.14-14-pve: immediate reset + CPU machine check when igb loads on I350 (Dell XC730xd) — 6.14 unaffected

Same issue on a Dell R630 (I350 rNDC, PVE 9.2)

pci=realloc=off on the kernel cmdline fixed it, so the igb blacklist is no longer needed

Likely cause: PVE kernels are built with CONFIG_PCI_REALLOC_ENABLE_AUTO=y

The kernel releases the BIOS-assigned bridge window and moves the I350 BARs
Code:
dmesg | grep "bridge window.*releasing"
The first register access from igb at the new address triggers a machine check.
 
Last edited:
Root cause found: PWR2262 / IERR reset on kernels >= 6.15 (patch posted, one-switch workaround)

I hit the same hard reset on a Dell PowerEdge R730: the iDRAC logs PWR2262 (Intel ME internal error), CPU0000 IERR and then a CPU machine check, right when igb probes the on-board I350. I bisected it on the hardware. Here's what I found, in case it helps others in this thread.

Cause: kernel commit 96336ec70264 ("PCI: Perform reset_resource() and build fail list in sync"), first released in 6.15. 6.14 is fine; 6.15 through current mainline (7.4-rc2) are affected.

The sequence:
  • With SR-IOV Global Enable off in the BIOS, the firmware doesn't assign SR-IOV space for the on-board 82599 10G NIC, so Linux turns on PCI resource reallocation by itself.
  • A few disabled option ROMs then fail to fit (harmless), but since 6.15 they count as failures.
  • The kernel re-lays out the PCI bridge windows and moves the chipset-attached I350.
  • When igb touches the I350 at its new address, the box resets. Sometimes the BIOS also halts in POST afterwards.

Workarounds, pick one:
  1. BIOS: Integrated Devices → SR-IOV Global Enable = Enabled. On my R730 this switch alone fixes it. I tested stock 6.17 with only SR-IOV enabled (boots) and with X2APIC + I/OAT enabled but SR-IOV off (still resets). The often-quoted X2APIC/I/OAT changes are harmless, but they aren't what fixes it.
  2. Kernel parameter pci=realloc=off. Stock 6.17 boots with it even with all three BIOS switches off.

Fix: a small patch that restores the pre-6.15 rule (disabled ROMs that don't fit aren't treated as failures) is posted to linux-pci: https://lore.kernel.org/linux-pci/?q=s:"PCI: Don't add failed disabled ROM BARs to fail_head"
It's tagged for stable, so once merged it should reach the LTS kernels the PVE kernel is based on. Tested on R730 (BIOS 2.19.0) with 6.17 and 7.4-rc2. Full logs and the bisect: https://bugzilla.kernel.org/show_bug.cgi?id=222253

If you have an affected R630/R730/T430/XC730 and can test the patch (or try pci=realloc=off and report whether it boots), replying on the linux-pci thread with a Tested-by: would help it land faster.
 
  • Like
Reactions: leesteken