e1000e eno1: Detected Hardware Unit Hang:

I just started looking into this issue and found here...testing the tso workaround.

I have three Dell Precision 3431s that use the Q246 C246 chipset. The first one I updated from PVE 8 to 9 started exhibiting this issue...more often when under network load like doing file transfers, but sometimes when it wasn't significant. The other two have not shown the problem...yet.

All are BIOS 1.32.0 (the latest when first deployed). 1.36.0 is out now, but details on fixes are pretty terse.
 
Last edited:
There is no "Q246" chipset, but only a "C246". And as mentioned, the original bug only should still affect older chipsets/systems (list see in my previous comment). So, if the issue persists and was not fixed with a BIOS update, then contact Dell. They should also be aware of this i219 Erratum.
 
celemine1gig said:
There is no "Q246" chipset, but only a "C246". And as mentioned, the original bug only should still affect older chipsets/systems (list see in my previous comment). So, if the issue persists and was not fixed with a BIOS update, then contact Dell. They should also be aware of this i219 Erratum.
Click to expand...

Oops, yes, that was a typo.
 
I have a few NUCs showing this behaviour, on 2 of them I have blacklisted the e1000e driver and use a different NIC which seems to work fine.

What I have noticed though, this only seems to happen on servers hosting VMs, I got one that only hosts LXCs which seems to be running forever and did migrate a VM across as I needed to perform some upgrades, as soon as the VM migrated across and started, the server crashed a few seconds/minutes later with the notorious log entry in the logs when I looked for a reason. When the server came back the upgrades had already completed and I migrated the VM away from the server back to the original hosting LCXs and VMs with the module being blacklisted, no issues whatsoever since.
 
Anyone know if the card below is affected by this bug? I have one laying around but not sure about the model, it says Intel PRO

WhatsApp Image 2026-01-09 at 1.44.52 PM.jpegWhatsApp Image 2026-01-09 at 1.44.53 PM.jpeg
 
I have just asked Claude

The reply was
Yes, almost certainly. Based on the Intel PRO/1000 PT Desktop Adapter identification, the chip under that heatsink is the Intel 82572GI controller, which uses the e1000e driver on Linux.
To confirm, if you have this card installed in a Linux system, you can run:

```
lspci | grep -i ethernet
```

Honestly unless you have high throughput requirements on the card I have seen no obvious impacts of the workaround and would do that
 
Last edited:
See Erratum Nr. 7: https://www.intel.com/content/dam/d...2571eb-82572ei-gbe-controller-spec-update.pdf
Just because it uses the "e1000e" driver does not necessarily mean, that it is affected by this bug.
You have to specifically check the Intel documentation on related topics. The problem might manifest itself slightly differently on those older chipsets. From what I saw, most poeple here saw it with i218/i219.
 
  • Like
Reactions: chrisn-au
glcarlos said:
One of the reasons why my setup is still pinned to kernel 6.8.12-8. I've had no issues since rolling back. Honestly have no desire to upgrade until this gets fixed.
Click to expand...

If I pin my kernel to 6.8.12-8 can I upgrade all the other packages that need upgrading? I've been holding off
 
Here’s what I did, which has been working for me so far — no network hangs yet.

First, I completely removed the community script (https://community-scripts.github.io/ProxmoxVE/scripts?id=nic-offloading-fix). This simply involved disabling the service and removing the service unit file. The script had actually been working perfectly fine for me, and I never experienced a hang while using it. I mainly removed it because I wanted to apply it manually then update and upgrade my machine. I figured that, worst case, I could downgrade and pin the kernel if needed.

I then applied the community script’s fix manually in /etc/network/interfaces using a post-up command. However, I only disabled TSO and applied it only to the physical Intel NIC — not the vmbr interface.

Code: 
iface enp0s25 inet manual
        post-up /sbin/ethtool -K $IFACE tso off

I made this change while running kernel 6.8.12-11-pve. After that, I performed a normal update and upgrade through the Proxmox web interface, which upgraded the system to 6.8.12-19-pve.

So far, everything has been running fine. I’ll report back if I experience any network hangs.

Code: 
admin@prox:~$ lspci | grep Eth
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05)
 
I could confirm, turning TSO off on phy nic (post-up script) fixed this issues for me as well (I219V), no hangs so far (2w), only slightly higher cpu utilization 1-2% during nic traffic
 
  • Like
Reactions: TheGadMan
In Ubiquiti networks, there's another problem. GSO/TSO is not the problem, but EEE (Energy Efficient Ethernet) is.

I've added something to /etc/network/interfaces which I see less mention of:

Code: 
# For network card vmbr0, the bridge for all real traffic
post-up ethtool -K vmbr0 tso off gso off

# For "nic0",  the physical network card:
post-up ethtool --set-eee nic0 eee off
post-up ethtool -K nic0 tso off gso off

Since I added that, my cluster has been free of crashes for three full weeks. Before that, it'd crash at least once a week.
 
After a year of running with "TSO OFF" (and all the others typically suggested) I'm still having issues. I don't get an obvious "hardware hang" message like others are reporting and the NIC will still report that it is up... but it is in fact not and this makes my bond rather useless as a result. After 90+ minutes of troubleshooting with LLMs the only thing I can narrow it down to is a "ring buffer" issue. The errors are higher on nodes with a higher workload.

`ethtool -S eno1 | grep -E "no_buffer|missed"` will show errors like so:
```
rx_no_buffer_count: 189
rx_missed_errors: 11696
```
Allegedly the solution is to increase the ring buffer size from 256 to 4096:
`ethtool -G eno1 rx 4096 tx 4096`

The final config appears as follows:
```
ethtool -g eno1
Ring parameters for eno1:
Pre-set maximums:
RX: 4096
RX Mini: n/a
RX Jumbo: n/a
TX: 4096
Current hardware settings:
RX: 4096
RX Mini: n/a
RX Jumbo: n/a
TX: 4096
RX Buf Len: n/a
CQE Size: n/a
TX Push: off
TCP data split: n/a
```

If I get failures after this then the only solution is to stop using the NIC entirely (or use it as the secondary with the USB NIC as primary).
 
I tried this morning to do an update to the new kernel 7.0, but again after some minutes the system became offline. SO when booting again with Linux 6.8.12-13-pve, the system remains online.

Do we still have no idea about the root cause why some kernel version lead the system to become unsstable while other are stable ?

My config is:
Lenovo ThinkCentre
AMD Ryzen 5 2400GE with Radeon Vega Graphics
64 GB RAM

LSPCI:
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus A
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus B
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 7
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0e)
01:00.1 Serial controller: Realtek Semiconductor Co., Ltd. RTL8111xP UART #1 (rev 0e)
01:00.2 Serial controller: Realtek Semiconductor Co., Ltd. RTL8111xP UART #2 (rev 0e)
01:00.3 IPMI Interface: Realtek Semiconductor Co., Ltd. RTL8111xP IPMI interface (rev 0e)
01:00.4 USB controller: Realtek Semiconductor Co., Ltd. RTL811x EHCI host controller (rev 0e)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] (rev c9)
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller
02:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
02:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
02:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
02:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Audio Coprocessor
02:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller
03:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 61)
 
Last edited:
wagnbeu0 said:
I tried this morning to do an update to the new kernel 7.0, but again after some minutes the system became offline. SO when booting again with Linux 6.8.12-13-pve, the system remains online.

Do we still have no idea about the root cause why some kernel version lead the system to become unsstable while other are stable ?

My config is:
Lenovo ThinkCentre
AMD Ryzen 5 2400GE with Radeon Vega Graphics
64 GB RAM

LSPCI:
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus A
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus B
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 7
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0e)
01:00.1 Serial controller: Realtek Semiconductor Co., Ltd. RTL8111xP UART #1 (rev 0e)
01:00.2 Serial controller: Realtek Semiconductor Co., Ltd. RTL8111xP UART #2 (rev 0e)
01:00.3 IPMI Interface: Realtek Semiconductor Co., Ltd. RTL8111xP IPMI interface (rev 0e)
01:00.4 USB controller: Realtek Semiconductor Co., Ltd. RTL811x EHCI host controller (rev 0e)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] (rev c9)
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller
02:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
02:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
02:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
02:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Audio Coprocessor
02:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller
03:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 61)
Click to expand...
Either you posted in the wrong thread, or you did not understand the topic at all.
This thread is about a very specific issue, that affects certain Intel Ethernet controllers.
However, in your pci listing I do not see one. So what exactly is it, that you are expecting to happen? Could it be, that you are looking for a completely different solution to your Realtek GBe related issue?
 
You must log in or register to reply here.
Top Bottom
Back