Kernel 7.0.14-14-pve: immediate reset + CPU machine check when igb loads on I350 (Dell XC730xd) — 6.14 unaffected

[Severenth]

Posting this because I have a fairly clean reproducer and I can't find an existing thread for it.

(please also forgive that I've been using Claude to troubleshoot this and summarise the post here)


Summary: on a Dell XC730xd-24 (the Nutanix XC appliance variant of the PowerEdge R730xd), kernel 7.0.14-14-pve resets the machine roughly 2.9 seconds into boot, at the moment the igb driver loads. iDRAC logs machine check errors against both CPUs. Kernel 6.14.11-9-pve on the same install is completely unaffected, and 7.0.14-14-pve boots and runs normally if I blacklist igb.

I'm aware the I350 is extremely common hardware, so if 7.0 broke igb generally this thread would already exist. I assume the trigger is narrower than "igb on I350" — possibly specific to this platform, this NVM revision, or dual-socket Haswell-EP. Happy to be told I've misread it.

Hardware​

• Dell XC730xd-24 (Nutanix XC variant of PowerEdge R730xd)
• BIOS 2.19.0, dated 12/12/2023 (latest available for this platform as far as I can tell)
• iDRAC8 Enterprise
• 2x Intel Xeon E5-2697 v3 (Haswell-EP), microcode revision 0x49
• 2x 4GB DDR4-2400 RDIMM (minimal config, more on order)
• PERC H730 in HBA mode
• Boot: ZFS RAID1 mirror on 2x SATA SSD, systemd-boot

Network daughter card (Intel X520/I350 combo), presenting as two devices on separate PCIe root ports:

01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
01:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
06:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
06:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)

I350 firmware, from ethtool -i eno3:
driver: igb
firmware-version: 1.67, 0x800010e0, 22.5.7
bus-info: 0000:06:00.0
Only the I350 half (igb) is involved in the failure. The 82599ES ports are unused — nothing is plugged into them.

Software​

• Proxmox VE 9.2.11, Debian 13 Trixie
• Broken: 7.0.14-14-pve
• Working: 6.14.11-9-pve
• Also worked previously: 6.8.12-43-pve under PVE 8.4

What happens​

Boot proceeds normally until:
[ 2.901179] igb: Intel(R) Gigabit Ethernet Network Driver
[ 2.901420] igb: Copyright (c) 2007-2014 Intel Corporation.

and the machine resets a moment later — fans ramp up and down, then a warm restart. Nothing reaches disk because journald hasn't started yet.

iDRAC SEL for the failed boot, within about three seconds:

A problem was detected related to the previous server boot.
CPU 1 machine check error detected. (Critical)
CPU 2 machine check error detected. (Critical)
CPU 1 machine check error detected. (Critical)
CPU 2 machine check error detected. (Critical)
CPU 1 machine check error detected. (Critical)
plus a large number of "An OEM diagnostic event occurred" entries. POST afterwards reports UEFI0078 (machine check errors in the previous boot).

Earlier attempts logged machine checks against CPU 1 only; the most recent hits both packages, which I read as a system-level error being reported by both rather than a per-CPU fault.

What I've tested​

Test	Result
6.8.12-43-pve (PVE 8.4)	Boots and runs fine
6.14.11-9-pve (PVE 9)	Boots and runs fine, igb binds both ports, no errors
7.0.14-14-pve	Reset at igb load, MCE on both CPUs
PVE 9.2 ISO installer (7.0.2-6-pve)	Same failure, same point
7.0.14-14-pve + modprobe.blacklist=igb	Boots fully, zero machine checks
One CPU physically removed	Same failure, same point
DIMMs swapped between sockets	No change
acpi=off	Installer can't see USB, dead end

Under 6.14 the boot log is clean: EDAC initialises all four memory controllers across both sockets with no errors, SMART on all 12 drives is clean, and there are no machine checks in normal operation.

With igb blacklisted under 7.0, grepping the boot log for machine checks, hardware errors and AER returns only routine AER-enable messages and the ACPI MCEJ table reservation — no error records, and no BERT entry carried over from the previous boot.

That combination is why I think this is an igb regression between 6.14 and 7.0 rather than failing hardware: the fault only appears when that one driver touches that one device, and the machine is otherwise stable.

Questions​

1. Is this already known, or has anything changed in igb between 6.14 and 7.0 that could produce a fatal PCIe/IIO error on an I350?
2. Is there anything more useful I can capture, given nothing survives to disk?
3. Is running 9.2 pinned to 6.14 a reasonable position to hold for now, or is there a workaround I've missed?

Happy to run any test you like — the box has no VMs on it yet, so I can break it freely.

Full diagnostic output attached.

 

[complexplaster27]

Does it boot if you install a plain vanilla Debian 13 on it? Does it even boot into the USB installer or does it do the same thing? I feel if you can't get to those points it's no longer a software problem.

For reference I'm running similar hardware to you XC640-10 CORE with 4x Intel(R) Ethernet 10G 4P X550/I350 rNDC and I don't have any of these types of issues at all.

 

[Severenth]

Does it boot if you install a plain vanilla Debian 13 on it? Does it even boot into the USB installer or does it do the same thing? I feel if you can't get to those points it's no longer a software problem.

For reference I'm running similar hardware to you XC640-10 CORE with 4x Intel(R) Ethernet 10G 4P X550/I350 rNDC and I don't have any of these types of issues at all.

Just installed Debian 13 netinst , installed without issue and network connected on the same config

however, once I try and install the latest PVE, it bows out at the same point


journal doesn't show anything, only the previous boot prior to the changes and attempt to run PVE

 

[leesteken]

Proxmox uses a kernel based on Ubuntu which is newer than the Debian one. Maybe there is an incompatibility between newer Linux kernels and your motherboard BIOS/UEFI. It often help to update the motherboard BIOS/UEFI. I have no experience with this but there are some threads about people having issues with the 7.0 kernels while 6.8, 6.14 or 6.17 worked fine. Maybe search for them and see if there are solutions or work-arounds.

 

[HHawk]

It seems quite similar to this bug: https://bugzilla.proxmox.com/show_bug.cgi?id=7969.
But I am no expert.

 

[Severenth]

Correction to my original post​

My first post framed this as a 7.0 problem. That was wrong, and I want to correct it before anyone spends time looking in the wrong place.

The regression is between 6.14 and 6.17. The 7.0 series inherited it rather than introducing it.

Full results, all on the same hardware:

6.8.12-43-pve            WORKS   (PVE 8.4, ZFS install)
6.12.107+deb13-amd64     WORKS   (Debian stock kernel, PVE userland installed)
6.14.11-9-pve            WORKS   (both installs)
6.17.13-21-pve           FAILS   (dies at igb, ~1.99s)
7.0.2-6-pve              FAILS   (PVE 9.2 ISO installer)
7.0.14-14-pve            FAILS   (dies at igb, ~2.9s)

6.14 and 6.17 have now both been tested on the same install, same userland, same packages — only the kernel differs. So this is not a Debian-versus-Proxmox userland question.

6.17 fails identically to 7.0, and recovers identically: modprobe.blacklist=igb lets both boot fully. Same driver, same instant, same workaround. I'm treating them as one bug.

Boot output at the failure point on 6.17.13-21-pve:

[    1.990632] ehci-pci 0000:00:1d.0: EHCI Host Controller
[    1.991170] ehci-pci 0000:00:1d.0: debug port 2
[    1.992574] igb: Intel(R) Gigabit Ethernet Network Driver
(one further igb line, then reset)

Also present earlier in that same boot, carried over from the previous crash:

[    1.636009] BERT: [Hardware Error]: Skipped 1 error records

Runtime reproducer​

More useful than the boot failure. Booted with modprobe.blacklist=igb, the system runs normally. Then from a login prompt:

modprobe igb

Instant lock and immediate reset. No boot process, no init ordering, nothing else in flight. The igb probe path alone triggers it.

Nothing survives to disk​

Configured journald with SyncIntervalSec=0 and ran journalctl --sync immediately before loading the module. After the reset:

• /sys/fs/pstore is empty
• the previous boot's journal ends on unrelated userspace noise — no igb, oops or machine check entries

Firmware resets the box before Linux can write anything. The iDRAC SEL appears to be the only record available short of serial-over-LAN capture.

@leesteken — BIOS​

Confirmed 2.19.0 (12/12/2023) is the latest available for this platform, checked against the service tag. Nothing newer offered, so a firmware update isn't available as a fix here.

Your point about the Proxmox kernel being Ubuntu-based was a useful steer — it's what prompted me to test Debian's own kernel, which works, and that in turn led to finding the real window.

@complexplaster27 — the XC640 comparison​

Worth noting the platforms differ more than the model numbers suggest. The XC640 is Skylake/Cascade Lake with an X550/I350 rNDC; mine is Haswell-EP with an X520/I350. Different CPU generation, different 10G silicon, different daughter card.

The I350 half is common to both, which is useful — it says I350 plus a recent kernel is fine on newer hardware, pointing at the platform rather than the NIC.

And to answer your original question directly: yes, plain Debian 13 installs and networks perfectly. It's specifically the newer kernels that fail.

@HHawk — re bug 7969​

Thanks for the pointer, that's a thorough report. Having checked, I don't think mine is the same issue:

The quirk never fires.Under 7.0.14-14-pve with igb blacklisted:

dmesg | grep -i "link speed restriction"

returns nothing. pcie_failed_link_retrain() isn't touching any port here.

Nothing is clamped for it to lift. The I350 sits behind PCH root port 00:1c.4, and Target Link Speed is 5GT/s, not clamped to 2.5GT/s. LnkSta matches LnkCap exactly — 5GT/s, Width x2, link fully trained. 7969's mechanism depends on a firmware 2.5GT/s clamp being lifted; there's no clamp here.


The device stays enumerated. Both I350 functions are present under the failing kernel:

06:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
06:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)

On the R610 the NIC disappears from the bus entirely.

The version boundary doesn't fit. 7969 identifies 7.0.14-7 as the regression point with -5 working. Mine fails from 6.17 onwards, well before that.

Same symptom family, different mechanism. Posting as a negative result so nobody applies that patch here expecting it to help.

Topology note

Possibly relevant to why the SEL blames both CPUs. The two halves of the rNDC reach the system by different routes:

00:03.0 -> bus 01 -> 82599ES  (CPU root port)
00:1c.4 -> bus 06 -> I350     (C610/X99 PCH root port)

Only the PCH-attached half misbehaves. A fatal PCIe error originating there traverses DMI2 before becoming a machine check, which may be why it surfaces against both packages rather than one.

Current state​

Running PVE 9.2 pinned to 6.14 and stable. 6.14 is now the newest kernel that works on this hardware.

Happy to run further tests, including any intermediate kernel or patched build — the box has no VMs on it and I can break it freely. If a 6.15 or 6.16 build is available anywhere, that would halve the window in one boot.

 

[Severenth]

Correction to my original post​

My first post framed this as a 7.0 problem. That was wrong, and I want to correct it before anyone spends time looking in the wrong place.

The regression is between 6.14 and 6.17. The 7.0 series inherited it rather than introducing it.

Full results, all on the same hardware:

6.8.12-43-pve            WORKS   (PVE 8.4, ZFS install)
6.12.107+deb13-amd64     WORKS   (Debian stock kernel, PVE userland installed)
6.14.11-9-pve            WORKS   (both installs)
6.17.13-21-pve           FAILS   (dies at igb, ~1.99s)
7.0.2-6-pve              FAILS   (PVE 9.2 ISO installer)
7.0.14-14-pve            FAILS   (dies at igb, ~2.9s)

6.14 and 6.17 have now both been tested on the same install, same userland, same packages — only the kernel differs. So this is not a Debian-versus-Proxmox userland question.

6.17 fails identically to 7.0, and recovers identically: modprobe.blacklist=igb lets both boot fully. Same driver, same instant, same workaround. I'm treating them as one bug.

Boot output at the failure point on 6.17.13-21-pve:

[    1.990632] ehci-pci 0000:00:1d.0: EHCI Host Controller
[    1.991170] ehci-pci 0000:00:1d.0: debug port 2
[    1.992574] igb: Intel(R) Gigabit Ethernet Network Driver
(one further igb line, then reset)

Also present earlier in that same boot, carried over from the previous crash:

[    1.636009] BERT: [Hardware Error]: Skipped 1 error records

Runtime reproducer​

More useful than the boot failure. Booted with modprobe.blacklist=igb, the system runs normally. Then from a login prompt:

modprobe igb

Instant lock and immediate reset. No boot process, no init ordering, nothing else in flight. The igb probe path alone triggers it.

Nothing survives to disk​

Configured journald with SyncIntervalSec=0 and ran journalctl --sync immediately before loading the module. After the reset:

• /sys/fs/pstore is empty
• the previous boot's journal ends on unrelated userspace noise — no igb, oops or machine check entries

Firmware resets the box before Linux can write anything. The iDRAC SEL appears to be the only record available short of serial-over-LAN capture.

@leesteken — BIOS​

Confirmed 2.19.0 (12/12/2023) is the latest available for this platform, checked against the service tag. Nothing newer offered, so a firmware update isn't available as a fix here.

Your point about the Proxmox kernel being Ubuntu-based was a useful steer — it's what prompted me to test Debian's own kernel, which works, and that in turn led to finding the real window.

@complexplaster27 — the XC640 comparison​

Worth noting the platforms differ more than the model numbers suggest. The XC640 is Skylake/Cascade Lake with an X550/I350 rNDC; mine is Haswell-EP with an X520/I350. Different CPU generation, different 10G silicon, different daughter card.

The I350 half is common to both, which is useful — it says I350 plus a recent kernel is fine on newer hardware, pointing at the platform rather than the NIC.

And to answer your original question directly: yes, plain Debian 13 installs and networks perfectly. It's specifically the newer kernels that fail.

@HHawk — re bug 7969​

Thanks for the pointer, that's a thorough report. Having checked, I don't think mine is the same issue:

The quirk never fires.Under 7.0.14-14-pve with igb blacklisted:

dmesg | grep -i "link speed restriction"

returns nothing. pcie_failed_link_retrain() isn't touching any port here.

Nothing is clamped for it to lift. The I350 sits behind PCH root port 00:1c.4, and Target Link Speed is 5GT/s, not clamped to 2.5GT/s. LnkSta matches LnkCap exactly — 5GT/s, Width x2, link fully trained. 7969's mechanism depends on a firmware 2.5GT/s clamp being lifted; there's no clamp here.


The device stays enumerated. Both I350 functions are present under the failing kernel:

06:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
06:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)

On the R610 the NIC disappears from the bus entirely.

The version boundary doesn't fit. 7969 identifies 7.0.14-7 as the regression point with -5 working. Mine fails from 6.17 onwards, well before that.

Same symptom family, different mechanism. Posting as a negative result so nobody applies that patch here expecting it to help.

Topology note

Possibly relevant to why the SEL blames both CPUs. The two halves of the rNDC reach the system by different routes:

00:03.0 -> bus 01 -> 82599ES  (CPU root port)
00:1c.4 -> bus 06 -> I350     (C610/X99 PCH root port)

Only the PCH-attached half misbehaves. A fatal PCIe error originating there traverses DMI2 before becoming a machine check, which may be why it surfaces against both packages rather than one.

Current state​

Running PVE 9.2 pinned to 6.14 and stable. 6.14 is now the newest kernel that works on this hardware.

Happy to run further tests, including any intermediate kernel or patched build — the box has no VMs on it and I can break it freely. If a 6.15 or 6.16 build is available anywhere, that would halve the window in one boot.

modinfo igb

srcversion 6.14 > 5A0CA79F58448BB84934CC50
srcversion 6.17 > 4D153E5BC4B4151DCEE7C61

was suggested these might help?

 

[complexplaster27]

So if installing Vanilla Debian works, does it stop working when you do this https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie ?
If it starts to fail might be worth raising a bug report with the proxmox team to resolve it if they can at all.

 

[Severenth]

I deviated in two ways — I installed a desktop environment and I kept the Debian kernels rather than removing them, both deliberately, as they're how I tested Debian's own kernel and kept networking while igb was blacklisted.

Neither though explain the big/issue: the failure is at 1.99s during PCI probe, before init runs. And the official ISO installer fails identically at the same line on the same hardware, with no Debian involved. The Debian-on-top install reproduced the bug rather than causing it.

 

[RobESI]

Seems like I have this problem on my one Dell R730 after upgrading to Proxmox 9.
Integrated NIC shows as: Intel(R) 2P X520/2P I350 rNDC in IDRAC.
Booting in 7.0.14-14 gives CPU machine check.
Switching back to 6.8.12-43-pve from previous version 8 seems to work fine.
Dell R720 works fine with this nic.

R730 CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Model 63 Stepping 2 (Haswell)
R720 CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz Model 62 Stepping 4 (Ivy Bridge)

 

[RobESI]

Fix:

• Reboot the server and press F2
• Navigate to System BIOS settings.
• Locate Processor Settings for the first item, then Integrated Devices
• Enable the following settings:
  • X2APIC Mode (Set to Enabled)
  • I/OAT DMA Engine / IOAT DMA (Set to Enabled)
  • SR-IOV Global (Set to Enabled, if available)
• Save / Reboot

 

[complexplaster27]

Sounds like you guys should really look at raising a ticket at https://bugzilla.proxmox.com/ highly doubt anyone on the forum has the capability to fix firmware issues at this point.

 

[RobESI]

Sounds like you guys should really look at raising a ticket at https://bugzilla.proxmox.com/ highly doubt anyone on the forum has the capability to fix firmware issues at this point.

I appreciate your feedback.

After researching these settings and the issue, it is possible these should have been enabled to begin with for a virtualization role. However, my personal best practices tell me to change the minimum settings from the default.
That said, the boot loop is a worst-case scenario for the unprepared administrator.

Respectfully, I request any interested parties advise the developers if they want.

AI slop below this line: ------

1. X2APIC Mode (Enabled)

• What it does: Expands processor interrupt handling, allowing the system to route hardware events across more than 255 CPU cores efficiently.
• Why it fixed the loop: Modern enterprise Linux kernels aggressively expect standard x2APIC mapping. If disabled, the kernel struggles with legacy interrupt remapping configurations or conflicts with BIOS flags, causing an immediate kernel panic on initialization.

2. SR-IOV Global (Enabled)

• What it does: Single-Root I/O Virtualization allows a single physical PCIe device (like a network card) to appear as multiple isolated virtual devices directly to the hypervisor.
• Why it fixed the loop: When the Linux kernel scans the PCIe bus, a partial or disabled SR-IOV topology mixed with advanced virtualization features can cause the PCI driver to map memory incorrectly, prompting a hardware exception.

3. I/OAT DMA Engine (Enabled)

• What it does: Offloads memory copy operations from the CPU to dedicated hardware channels.
• Why it fixed the loop: While generally safe to leave off on normal setups, the specific Linux kernel versions bundled with early Proxmox 9 setups require a fully cohesive IOMMU environment. Enabling I/OAT pairs with SR-IOV to provide the exact direct memory access layout the kernel needs to securely register enterprise network cards during boot.

By turning all three On, you provided a complete, unified hardware virtualization stack. Proxmox 9's newer kernel was able to map all system memory lanes, CPU interrupts, and PCIe virtual interfaces flawlessly without encountering the device initialization crash.