e1000 driver hang

sdet00 said:
I can't believe this is STILL a problem in 2024!
Click to expand...
Even in 2025.
Hang yesterday w. proxmox-kernel-6.8.12-4-pve-signed
Log AFTER reboot.
Bash: 
# lspci -nnk
...
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) [B]I219-LM [8086:15bb] (rev 10)[/B]
        DeviceName: Onboard - Ethernet
        Subsystem: ASUSTeK Computer Inc. Ethernet Connection (7) I219-LM [1043:8686]
        Kernel driver in use: e1000e
        Kernel modules: e1000e
...

# journalctl -b 0 | grep e1000e
Jan 12 1X:YY:ZZ example kernel: e1000e: Intel(R) PRO/1000 Network Driver
Jan 12 1X:YY:ZZ example kernel: e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
Jan 12 1X:YY:ZZ example kernel: e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
Jan 12 1X:YY:ZZ example kernel: e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
Jan 12 1X:YY:ZZ example kernel: e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 04:d9:f5:bc:35:1a
Jan 12 1X:YY:ZZ example kernel: e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
Jan 12 1X:YY:ZZ example kernel: e1000e 0000:00:1f.6 eth0: MAC: 13, PHY: 12, PBA No: FFFFFF-0FF
Jan 12 1X:YY:ZZ example kernel: e1000e 0000:00:1f.6 eno1: renamed from eth0
Jan 12 1X:YY:ZZ example kernel: e1000e 0000:00:1f.6 eno1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
# comment added: see hang
Jan 12 22:02:03 example kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
Jan 12 22:02:05 example kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
Jan 12 22:02:07 example kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
Jan 12 22:02:09 example kernel: e1000e 0000:00:1f.6 eno1: NETDEV WATCHDOG: CPU: 6: transmit queue 0 timed out 5251 ms
Jan 12 22:02:09 example kernel: e1000e 0000:00:1f.6 eno1: Reset adapter unexpectedly
Jan 12 22:02:13 example kernel: e1000e 0000:00:1f.6 eno1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
proxmox-kernel-6.8.12-5-pve-signed is installed now.

Followed https://forum.proxmox.com/threads/e1000-driver-hang.58284/page-11#post-714192 now.
 
Last edited:
Same here, in 2 of my servers work around seems working, but in one my servers it is not working. Is there any permanent solution. Did anyone tried to upgrade to latest version of proxmox? Is this problem relevant to debian based systems? If yes, unfortunately I have started to think to find another solution to host vms. I have thought buying an external network card, but I am not sure I will meet the same problem or not. Have anyone tried?
 
I'm also experiencing "Detected Hardware Unit Hang" on my X11SCA-F with Linux 6.12 on debian. Disabling TSO with `ethtool -K tso off` should fixed the issue.

Looking into the kernel code, the TARC0 register in i225 BAR0 is probably related to the issue. But it was never documented by Intel.
C: 
// https://github.com/torvalds/linux/blob/febbc555cf0fff895546ddb8ba2c9a523692fb55/drivers/net/ethernet/intel/e1000e/netdev.c#L3000
/* SPT and KBL Si errata workaround to avoid data corruption */
if (hw->mac.type == e1000_pch_spt) {
    u32 reg_val;

    reg_val = er32(IOSFPC);    // 0x0F28
    reg_val |= E1000_RCTL_RDMTS_HEX;    // 0x00010000
    ew32(IOSFPC, reg_val);

    reg_val = er32(TARC(0));    // 0x3840
    /* SPT and KBL Si errata workaround to avoid Tx hang.
     * Dropping the number of outstanding requests from
     * 3 to 2 in order to avoid a buffer overrun.
     */
    reg_val &= ~E1000_TARC0_CB_MULTIQ_3_REQ;    // 0x30000000
    reg_val |= E1000_TARC0_CB_MULTIQ_2_REQ;     // 0x20000000
    ew32(TARC(0), reg_val);
}
The code was introduced in https://github.com/torvalds/linux/commit/b10effb92e272051dd1ec0d7be56bf9ca85ab927 , but I was unable to find the definition of the IOSFPC register and TARC register. :(
 
  • Like
Reactions: Licht
Although it is defined as below, I am still facing with this error repeatedly, is there any problem with below definitions?

Code: 
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
        post-up ethtool -K eno1 gso off gro off tso off tx off rx off
        dns-nameservers 8.8.8.8 8.8.4.4

auto vmbr0
iface vmbr0 inet static
        address 192.168.20.50/24
        gateway 192.168.20.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        post-up ethtool -K vmbr0 gso off gro off tso off tx off rx off
 
Just got this error after the latest (non-paid) update on my Intel NUC with PVE. Gonna try the ethtool workaround.
 
I thought I fixed this by updated /etc/network/interfaces with a section like this:

```
iface eno2 inet manual
offload-gso off
offload-gro off
offload-tso off
offload-rx off
offload-tx off
offload-rxvlan off
offload-txvlan off
offload-sg off
offload-ufo off
offload-lro off
```

However, after running ` ethtool -k eno2 | grep offload` I noticed that `rx-vlan-offload` and `tx-vlan-offload` where still `on`. I've been trying to find the documentation for /etc/network/interfaces so verify the `offload-rxvlan` setting and I found this: https://salsa.debian.org/kernel-team/ethtool/-/blob/debian/latest/debian/README.Debian There is no `offload-rxvlan` or `offload-txvlan` in that doc, so maybe the syntax is different, or the vlan offload options are just available in /etc/network/interfaces. If anyone has more documentation that would be great.

I noticed a lot of others thought they applied the workaround but are still having issues, for them, make sure to use `ethtool -k` to verify all offloading is off.
 
  • Like
Reactions: SteveTKO
Hi team ,

Still found hang also on Intel 1GbE LAN ports by Intel i210 , will try
#ethtool -K <interface> tso off gso off
 
Hi, I faced this issue in the last few days, after months of stable uptime. I added the line "post-up ethtool -K eno1 tso off gso off" to my /etc/network/interfaces config. I will report in the next days if problem persist.
 
  • Like
Reactions: golfreeze
pomp92 said:
Hi, I faced this issue in the last few days, after months of stable uptime. I added the line "post-up ethtool -K eno1 tso off gso off" to my /etc/network/interfaces config. I will report in the next days if problem persist.
Click to expand...
what kernel version and proxmox version ?
 
proxmox-ve: 8.3.0 (running kernel: 6.8.12-9-pve). Hardware is a Lenovo m720q so Intel I219-V.
Few days ago I installed the last free updates that probably started the issue, but also installed NUT, added a qdevice to my cluster and replaced a network switch. My other server that's running same proxmox and kernel versions uses one intel i226-V NIC and it's working fine.
 
  • Like
Reactions: golfreeze
I started getting this problem on Dell Optiplex 9020 devices after upgrading from proxmox 7.4 to 8.3
Just trying the suggested fix now.
 
Same here, upgrading from proxmox 7.4 to 8.3 the other day. Started noticing this also. I could unplug the network cable and plug it back in and everything started working. I thought I had a bad cable so I replaced it. Problem still happened so I changed switch ports. Still having issue and ended up here.

I too will try adding the:

post-up ethtool -K gso off tso off rxvlan off txvlan off gro off tx off rx off sg off

to my interfaces file and report back.

Thankful for the community!
 
Hi, I've had the same problem since the last update on Thursday. Never had this problem before. For me, it helps if I just unplug and replug the network cable from the NUC to the switch.

Since I'm unfortunately not 100% sure about Proxmox yet, I'd like to ask again. Is it sufficient to add the line
Code: 
post-up ethtool -K gso off tso off rxvlan off txvlan off gro off tx off rx off sg off
in my interfaces file, or do I need to install something else first?

And for me to learn, what does this Codeline do?

Thanks, and sorry for the stupid questions.
 
Last edited:
pomp92 said:
proxmox-ve: 8.3.0 (running kernel: 6.8.12-9-pve). Hardware is a Lenovo m720q so Intel I219-V.
Few days ago I installed the last free updates that probably started the issue, but also installed NUT, added a qdevice to my cluster and replaced a network switch. My other server that's running same proxmox and kernel versions uses one intel i226-V NIC and it's working fine.
Click to expand...
Ok noted for your information , Let 's monitoring on this workaround for 1 week.
 
Gorki said:
Hi, I've had the same problem since the last update on Thursday. Never had this problem before. For me, it helps if I just unplug and replug the network cable from the NUC to the switch.

Since I'm unfortunately not 100% sure about Proxmox yet, I'd like to ask again. Is it sufficient to add the line
Code: 
post-up ethtool -K gso off tso off rxvlan off txvlan off gro off tx off rx off sg off
in my interfaces file, or do I need to install something else first?

And for me to learn, what does this Codeline do?

Thanks, and sorry for the stupid questions.
Click to expand...
Hi Gorki,

Let me share some information , base on my understanding
"post-up ethtool -K interface_name gso off tso off rxvlan off txvlan off gro off tx off rx off sg off"
this could added into "/etc/network/interface" file such as

auto vmbr0
iface vmbr0 inet static
address 103.x.y.z/24
gateway 103.x.y.xx
bridge-ports enp0
bridge-stp off
bridge-fd 0
post-up ethtool -K vmbr0 tso off gso off gro off

and will effect every time that server reboot or called "persistent on"
or Tells the system to execute the command after the vmbr0 interface is activated.

== However command below is command you could run on shell directly and effect only 1 times
#ethtool -K <interface> tso off gso off
Just run as manually , when server reboot will rollback setting.

== what meaning of each value.
TSO (TCP Segmentation Offload) : when setting to "off" meaning is NIC segments packets, may interfere with packet filters
GSO (Generic Segmentation Offload) : when setting to "off" meaning is Same as TSO, at software layer,
GRO (Generic Receive Offload) : when setting to "off" meaning is Merges packets before they hit the kernel — bad for sniffing, firewalls
 
Last edited:
Gorki said:
Hi, I've had the same problem since the last update on Thursday. Never had this problem before. For me, it helps if I just unplug and replug the network cable from the NUC to the switch.

Since I'm unfortunately not 100% sure about Proxmox yet, I'd like to ask again. Is it sufficient to add the line
Code: 
post-up ethtool -K gso off tso off rxvlan off txvlan off gro off tx off rx off sg off
in my interfaces file, or do I need to install something else first?

And for me to learn, what does this Codeline do?

Thanks, and sorry for the stupid questions.
Click to expand...
I added the setting by running the command as root at the console. "ethtool -K eno1 gso off tso off rxvlan off txvlan off gro off tx off rx off sg off" it has been 24hr and the problem has NOT happened again. So today I added "post-up ethtool -K gso off tso off rxvlan off txvlan off gro off tx off rx off sg off" to my interfaces file. So it will enable again at reboot. Hopefully the issue will be resolved in the next update. Right now it is working and I have not noticed any bad side effects.

Thanks.
 
Last edited:
Thanks for your feedbacks and clarification.
One more question from my side. What is the correct command now?

post-up ethtool -K interface_name gso off tso off rxvlan off txvlan off gro off tx off rx off sg off (with interface_name)
or
post-up ethtool -K gso off tso off rxvlan off txvlan off gro off tx off rx off sg off (without interface_name)

And if I need interface_name, where can I find it?
 
Last edited:
Gorki said:
Thanks for your feedbacks and clarification.
One more question from my side. What is the correct command now?

post-up ethtool -K interface_name gso off tso off rxvlan off txvlan off gro off tx off rx off sg off (with interface_name)
or
post-up ethtool -K gso off tso off rxvlan off txvlan off gro off tx off rx off sg off (without interface_name)

And if I need interface_name, where can I find it?
Click to expand...
Your absolutely right you need to tell the adapter interface in the command. I missed that when typing my post sorry.

ethtool -K eno1 gso off tso off rxvlan off txvlan off gro off tx off rx off sg offpost-up ethtool -K gso off tso off rxvlan off txvlan off gro off tx off rx off sg off

To find my interface name i did a cat /etc/network/interfaces

Look for line that says "iface" en###"


For editing the interfaces file good look at some of the earliest posts.

Hope this helps you.
 
  • Like
Reactions: bubblegoose
You must log in or register to reply here.
Top Bottom