e1000 driver hang

Adding another datapoint.

Dell T5810 with a

00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05)

Been fine for more than a year, with Linux version 6.8.4-2-pve but upgraded mid-March, and ~9 days later had the first e1001e Detected Hardware Unit Hang error - that is with Linux version 6.8.12-20-pve

I didn't investigate it at the time, just assumed a random crash, but then again another ~ 9 days later and same thing.

I've setup a cron entry - but it's pretty ugly - just tries to ping my router, and if that fails twice in a row at 15m interval, force a reboot:

Code: 
*/15 * * * * root ping -c3 -W2 192.168.1.1 > /dev/null 2>&1 && rm -f /run/ping-watchdog-fail || { [ -f /run/ping-watchdog-fail ] && /usr/sbin/shutdown -r now || touch /run/ping-watchdog-fail; }

This combined with proxmox-boot-tool kernel pin 6.8.4-2-pve means I should only have one more failure with 6.8.12-* branch.

At some point I'll have to revisit newer proxmox kernel releases to see if the regression's been properly fixed.
 
macamba said:
Post update:
Upgraded to Proxmox 9 and still NiCS crashes. This time with hardware failure in log. Adding post-up to interface in /etc/network/interfaces -suggested already in this thread- seems to work for me:
Code: 
iface nic0 inet manual
        post-up /sbin/ethtool -K nic0 gso off gro off tso off

Original post:
I have two Intel NiCS on the following Z390M-ITX/ac mainboard:
- Gigabit LAN 10/100/1000 Mb/s
- 1 x Giga PHY Intel® I219V, 1 x GigaLAN Intel® I211AT
Tested confirmed that both crash under load. Crashing occured after upgrading to latest Proxmox 8 kernel. Before this kernel sometimes remote connection problems which I assume now after words already had to do with this issue. Would upgrading to Proxmox 9 help?
Click to expand...
Above config kept the NIC crashes away for a couple of months, but they came back with no trace in the log. After adding 'tx off' and 'rx off' to the config as well it seems things are stable again. For easy reference:
Code: 
iface nic0 inet manual
        post-up /sbin/ethtool -K nic0 gso off gro off tso off tx off rx off
 
So what's the current status on this? I had a bit of drama this morning because of another crash due to "kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang". One of my nodes experiences this roughly every week.

I'm still on 8.4, are there any fixes/workarounds for the network driver crash? If there aren't, proxmox 9 doesn't seem to be fixed either. I was looking for a good reason to finally migrate to 9, but I'm afraid this isn't it.
 
@kokoon what is your exact current running kernel version? I do ask, as here I saw similar issues with unstable network and SAS since few days, after updating from `6.8.12-22-pve` to `6.8.12-28-pve` on several nodes. We're still in the middle of investigation, which means this is just a guess so far that the issues might be related to a recent kernel update.
 
Since this problem exists now for many years, I expect no fix for the Hardware Unit Hang on e1000 driver anymore.
I configured all nodes with this problem not to use the offloading feature in the network interfaces file.
 
Lutz Willek said:
@kokoon what is your exact current running kernel version? I do ask, as here I saw similar issues with unstable network and SAS since few days, after updating from `6.8.12-22-pve` to `6.8.12-28-pve` on several nodes. We're still in the middle of investigation, which means this is just a guess so far that the issues might be related to a recent kernel update.
Click to expand...
6.8.12-29-pve
 
TomFIT said:
Since this problem exists now for many years, I expect no fix for the Hardware Unit Hang on e1000 driver anymore.
I configured all nodes with this problem not to use the offloading feature in the network interfaces file.
Click to expand...
So disabling offloading is a definite workaround for it, I gather. That's good enough for me, just a homelab in question here.
 
Had this happen on 8.4.19 with kernel 6.8.12-23-pve tonight, with offloads verified disabled.
 
Adding another datapoint:

Machine: HP EliteDesk 800 G6 Mini
Actions taken: disabled TSO, GSO, GRO and EEE
Proxmox version: 9.2.0 (running kernel: 7.0.14-9-pve).
Outcome: While the E1000 NIC doesn't take down the entire server now, it does randomly drop the VM on the PVE host, when streaming 4k from Jellyfin.

Pretty disappointed to be honest, maybe there's something I missed but after days of tests and troubleshooting I gave up and brought a 2.5gb flex io Ethernet expansion card
 
Last edited:
I've upgraded my system to PVE 9.2.x about 2 months ago. I had the offloading turned off and yet this week the network froze. Even pulling the ethernet and putting it back did not help this time. I had to restart the computer. Before the upgrade I had the 6.8 kernel pined.

I also had a watch script that I found here, only to discover that it had a logical error and was not working. I did some LLM magic with it and got it working now. This version also disables the power management on the NIC, as in I found out in windows that disabling it resolves a lot of network related issues.
So far there are no entries in the log, so this last freeze might be a really random occurrence, I don't know, but I keep monitoring the system.

(I don't want to post the script here until I know it's working, but if someone really wants it, DM me.)
 
Last edited:
Just a heads-up regarding an upcoming kernel update. Emphasis mine.

proxmox-kernel-7.0 (7.0.14-28) trixie; urgency=medium

* ZFS: actually revert the fix for memory-mapped files serving stale data
after a rollback or receive (#7653). The 7.0.14-27 package still
contained it, as its build reused outdated ZFS sources, so receives and
rollbacks into mounted datasets could still deadlock as described there.

* fix #8264: net: fix a kernel crash when creating an ipip tunnel, which
hung all networking on the host until a reset since 7.0.14-25. Root in
an unprivileged container could trigger it too.

* amdgpu: stop logging a harmless warning with a stack trace on every boot
of some systems with AMD APUs, for example on ASRock Rack server boards,
whose firmware lists display outputs without hot-plug detection.

* e1000e: work around a hardware erratum that made Intel I219 NICs send
corrupted data and then hang ("Detected Hardware Unit Hang") with TCP
segmentation offload, triggered for example by forwarding VM traffic
over a bridge. Disabling TSO as a workaround should no longer be
needed.

* overlayfs: fix use-after-free bugs, a crash and an endless loop that
unprivileged users, also root in unprivileged containers, could trigger
with their own overlay mounts, and keep a change through an open
descriptor of a removed file from replacing whatever took its name in the
upper layer.

* fix a use-after-free in case-insensitive lookups on filesystems with
casefolding enabled, and races in io_uring on exec and coredumps,
including a use-after-free, that local users could trigger.

* net: fix an endless loop in SCTP that stalls a CPU, which a remote SCTP
peer or a local user could trigger, and memory corruption in VM sockets
(vsock). Also charge AF_XDP ring buffers to the memory cgroup that
creates them and bound an ebtables allocation, so that containers cannot
use these to exhaust the memory of the host.

-- Proxmox Support Team <support@proxmox.com> Mon, 02 Nov 2026 12:54:27 +0100
Click to expand...

proxmox-kernel-7.0_7.0.14-28.changelog

I think the issue is finally (going to be) over, folks.
 
  • Like
Reactions: coq, Barry69 and Christoph Lechleitner
You must log in or register to reply here.
Top Bottom
Back