[SOLVED] Intel NIC e1000e hardware unit hang

sloppyjoe · Oct 19, 2024

the_Uli said:
that seems to have fixed it for me.
I have no performance problems and no more problems are visible in the log since then:

Code:

ethtool -K eth0 gso off gro off tso off tx off rx off rxvlan off txvlan off

Really appreciate the help with this as the constant disconnects were driving me crazy!

This fixed my issues with the Intel i219 adapter in my Lenovo M920q.

It seems that this fix needs to be applied every time you restart. So remember to set it up in systemctl or however you run your startup scripts.

Code:

lspci | grep -i ethernet
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)

RedNano · Dec 31, 2024

Hi all!! I just moved from ESXi yesterday and I'm noticing this same error on my NUC10FNH.

Does applying "ethtool -K eth0 gso off gro off tso off tx off rx off rxvlan off txvlan off" solve the issue?
Should all vms be powered off to apply this?

Sunrise · Jan 2, 2025

I have the same problem. Proxmox runs on an old laptop.
The error appears every 1,5 - 6 weeks. Only restarting helps.
Ideas?

RedNano · Jan 31, 2025

Hi!!!
has this been solved in recent proxmox patches?

parallax · Feb 5, 2025

I have the same issue on one of my Proxmox servers.
BUT i have exactly the same server running Proxmox without any issues.
Both newly installed, Proxmox up to date, same BIOS version, same offloading settings,...

If someone can point me in the right direction to debug this, it seems a very good usecase as i have 2 servers with identical hardware.

TErxleben · Feb 20, 2025

parallax said:
I have the same issue on one of my Proxmox servers.
BUT i have exactly the same server running Proxmox without any issues.
Both newly installed, Proxmox up to date, same BIOS version, same offloading settings,...

If someone can point me in the right direction to debug this, it seems a very good usecase as i have 2 servers with identical hardware.

I have the exact same problem here. I have now made the modification

ethtool -K vmbr0 gso off gro off tso off tx off rx off rxvlan off txvlan of
ethtool -K eno1 gso off gro off tso off tx off rx off rxvlan off txvlan of

on the Proxmoxhost and will report further.

chris8837 · Feb 25, 2025

sloppyjoe said:
Really appreciate the help with this as the constant disconnects were driving me crazy!

This fixed my issues with the Intel i219 adapter in my Lenovo M920q.

It seems that this fix needs to be applied every time you restart. So remember to set it up in systemctl or however you run your startup scripts.

Code:

lspci | grep -i ethernet 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)

Do you have a turorial on how to run this everytime it reboots? New to all this

rocha.best · Apr 2, 2025

Same issue here,

Starts with 8.3.0 PVE version, only on heavy-load (abobe 60/70% throughput of 1GbE onboard interface (Intel I219-LM).

On 8.3.4 PVE version, few days after update, we completely lost connection at night.
Host rebooted, connectivity ok.
Next day, same issue.
Update to 8.3.5 - no changes at all.

Since yesterday (2025-04-01), after few hours working, it is dead again.
{ PVE current Version - pve-manager/8.3.5/dac3aa88bac3f300 (running kernel: 6.8.12-9-pve) }

Syslog shows the same error message over and over again.

kernel: e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:

Today, from 8:00AM to 10:30AM, more than 4300 records from message above.

Server rebooted again, 2,5 hours working, same issue again.

After this last reboot, used this command to check flags...

/sbin/ethtool -k enp0s31f6

And this to set flags...
(run at 2025-04-02@17:03 (UTC))

/sbin/ethtool -K enp0s31f6 tx off rx off

I have tried to edit the /etc/network/if-up.d/ethtool2 file, adding "/sbin/ethtool -K enp0s31f6 tx off rx off" on it, NO SUCCESS.

So, I have edited the /etc/network/interfaces file, add some parameters on file, as follows.

##/etc/network/interfaces file
auto lo
iface lo inet loopback

auto enp0s31f6
iface enp0s31f6 inet manual
post-up /sbin/ethtool -K enp0s31f6 tx off rx off

auto vmbr0
iface vmbr0 inet static
address 10.20.30.36/29
gateway 10.20.30.33
bridge-ports enp0s31f6
bridge-stp off
bridge-fd 0

iface wlp0s20f3 inet manual

auto vmbr1
iface vmbr1 inet static
address 192.168.1.100/24
bridge-ports none
bridge-stp off
bridge-fd 0

auto vmbr2
iface vmbr2 inet static
address 192.168.2.0/24
bridge-ports none
bridge-stp off
bridge-fd 0

## end of interfaces file

iface enp0s31f6 inet manual
post-up /sbin/ethtool -K enp0s31f6 tx off rx off

Finally, after reboot, I have checked the "results"

"
root@hostname:~# /sbin/ethtool -k enp0s31f6
Features for enp0s31f6:
rx-checksumming: off
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off [requested on]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
"

----------------------------------
2025/04/11@18:35 (UTC)
It works fine since configuration - enven when reboot PVE
updated to 8.4.1 - no issues at all

rocha.best · Apr 11, 2025

chris8837 said:
Do you have a turorial on how to run this everytime it reboots? New to all this

Please check my last message - it is working here

Pete_79 · Apr 12, 2025

rocha.best said:
Same issue here,

Starts with 8.3.0 PVE version, only on heavy-load (abobe 60/70% throughput of 1GbE onboard interface (Intel I219-LM).

On 8.3.4 PVE version, few days after update, we completely lost connection at night.
Host rebooted, connectivity ok.
Next day, same issue.
Update to 8.3.5 - no changes at all.

Since yesterday (2025-04-01), after few hours working, it is dead again.
{ PVE current Version - pve-manager/8.3.5/dac3aa88bac3f300 (running kernel: 6.8.12-9-pve) }

Syslog shows the same error message over and over again.

kernel: e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:

Today, from 8:00AM to 10:30AM, more than 4300 records from message above.

Server rebooted again, 2,5 hours working, same issue again.

After this last reboot, used this command to check flags...

/sbin/ethtool -k enp0s31f6

And this to set flags...
(run at 2025-04-02@17:03 (UTC))

/sbin/ethtool -K enp0s31f6 tx off rx off

I have tried to edit the /etc/network/if-up.d/ethtool2 file, adding "/sbin/ethtool -K enp0s31f6 tx off rx off" on it, NO SUCCESS.

So, I have edited the /etc/network/interfaces file, add some parameters on file, as follows.

iface enp0s31f6 inet manual
post-up /sbin/ethtool -K enp0s31f6 tx off rx off

Finally, I have checked the "results"

"
root@hostname:~# /sbin/ethtool -k enp0s31f6
Features for enp0s31f6:
rx-checksumming: off
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off [requested on]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
"

----------------------------------
2025/04/11@18:35 (UTC)
It works fine since configuration - enven when reboot PVE
updated to 8.4.1 - no issues at all

What if you have a bridge?
Add the post-up stuff on both the nic and the bridge?

muzicman0 · Apr 13, 2025

Pete_79 said:
What if you have a bridge?
Add the post-up stuff on both the nic and the bridge?

Just the NIC.

Pete_79 · Apr 13, 2025

muzicman0 said:
Just the NIC.

Thanks, i did the changes yesterday, but today its reverted for some reason. Is this file affected from corosync or someting? i have a 3 node clsuter.

muzicman0 · Apr 13, 2025

I have a cluster as well with no issues. Are you sure you saved it?

Pete_79 · Apr 13, 2025

muzicman0 said:
I have a cluster as well with no issues. Are you sure you saved it?

Thats a good question, i was tired yesterday evening. I did the changes again and i'll see tomorrow. I have expansion NICs that i am running on now, that works liek it should. Would be nice to free up the PCIe slot tho if i get this to work.

valr30 · Apr 14, 2025

I also have that problem on 8.4.1 . i also applied the patch as said :
my /etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno2 inet manual
offload-gso off
offload-gro off
offload-tso off
offload-rx off
offload-tx off
offload-rxvlan off
offload-txvlan off
offload-sg off
offload-ufo off
offload-lro off
auto vmbr0
iface vmbr0 inet static
address 192.168.1.4/24
gateway 192.168.1.1
bridge-ports eno2
bridge-stp off
bridge-fd 0
offload-gso off
offload-gro off
offload-tso off
offload-rx off
offload-tx off
offload-rxvlan off
offload-txvlan off
offload-sg off
offload-ufo off
offload-lro off

iface wlo1 inet manual

the log :
Apr 13 21:28:12 vrProxmox pvedaemon[154583]: <root@pam> successful auth for user 'root@pam'
Apr 13 21:35:04 vrProxmox pveproxy[156592]: worker exit
Apr 13 21:35:05 vrProxmox pveproxy[1219]: worker 156592 finished
Apr 13 21:35:05 vrProxmox pveproxy[1219]: starting 1 worker(s)
Apr 13 21:35:05 vrProxmox pveproxy[1219]: worker 209836 started
Apr 13 21:35:22 vrProxmox kernel: e1000e 0000:00:1f.6 eno2: Detected Hardware Unit Hang: TDH <8> TDT <56> next_to_use <56> next_to_clean <8>buffer_info[next_to_clean]: time_stamp <103160197> next_to_watch <9> jiffies <103160cc0> next_to_watch.status <0>MAC Status <40080083>PHY Status <796d>PHY 1000BASE-T Status <3800>PHY Extended Status <3000>PCI Status <10>
Apr 13 21:35:24 vrProxmox kernel: e1000e 0000:00:1f.6 eno2: Detected Hardware Unit Hang: TDH <8> TDT <56> next_to_use <56> next_to_clean <8>buffer_info[next_to_clean]: time_stamp <103160197> next_to_watch <9> jiffies <103161480> next_to_watch.status <0>MAC Status <40080083>PHY Status <796d>PHY 1000BASE-T Status <3800>PHY Extended Status <3000>PCI Status <10>
Apr 13 21:35:26 vrProxmox kernel: e1000e 0000:00:1f.6 eno2: Detected Hardware Unit Hang: TDH <8> TDT <56> next_to_use <56> next_to_clean <8>buffer_info[next_to_clean]: time_stamp <103160197> next_to_watch <9> jiffies <103161c80> next_to_watch.status <0>MAC Status <40080083>PHY Status <796d>PHY 1000BASE-T Status <3800>PHY Extended Status <3000>PCI Status <10>

and it continue like that until you reboot

i am now trying the solution from sudsy from here https://forum.proxmox.com/threads/system-loses-networking.98705/
i also have a VLAN tag from my router, maybe thats the problem.

and added this to the /etc/network/interfaces:

auto vmbr0.40
iface vmbr0.40 inet static
address 192.168.1.4/24
gateway 192.168.1.1

rocha.best · Apr 14, 2025

Pete_79 said:
What if you have a bridge?
Add the post-up stuff on both the nic and the bridge?

Here is my /etc/network/interfaces file
I have just added post-up on NIC. The vmbr0 uses this NIC, other bridges does not use it.

##/etc/network/interfaces file
auto lo
iface lo inet loopback

auto enp0s31f6
iface enp0s31f6 inet manual
post-up /sbin/ethtool -K enp0s31f6 tx off rx off

auto vmbr0
iface vmbr0 inet static
address 10.20.30.36/29
gateway 10.20.30.33
bridge-ports enp0s31f6
bridge-stp off
bridge-fd 0

iface wlp0s20f3 inet manual

auto vmbr1
iface vmbr1 inet static
address 192.168.1.100/24
bridge-ports none
bridge-stp off
bridge-fd 0

auto vmbr2
iface vmbr2 inet static
address 192.168.2.0/24
bridge-ports none
bridge-stp off
bridge-fd 0

valr30 · Apr 15, 2025

valr30 said:
I also have that problem on 8.4.1 . i also applied the patch as said :
my /etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno2 inet manual
offload-gso off
offload-gro off
offload-tso off
offload-rx off
offload-tx off
offload-rxvlan off
offload-txvlan off
offload-sg off
offload-ufo off
offload-lro off
auto vmbr0
iface vmbr0 inet static
address 192.168.1.4/24
gateway 192.168.1.1
bridge-ports eno2
bridge-stp off
bridge-fd 0
offload-gso off
offload-gro off
offload-tso off
offload-rx off
offload-tx off
offload-rxvlan off
offload-txvlan off
offload-sg off
offload-ufo off
offload-lro off

iface wlo1 inet manual

the log :
Apr 13 21:28:12 vrProxmox pvedaemon[154583]: <root@pam> successful auth for user 'root@pam'
Apr 13 21:35:04 vrProxmox pveproxy[156592]: worker exit
Apr 13 21:35:05 vrProxmox pveproxy[1219]: worker 156592 finished
Apr 13 21:35:05 vrProxmox pveproxy[1219]: starting 1 worker(s)
Apr 13 21:35:05 vrProxmox pveproxy[1219]: worker 209836 started
Apr 13 21:35:22 vrProxmox kernel: e1000e 0000:00:1f.6 eno2: Detected Hardware Unit Hang: TDH <8> TDT <56> next_to_use <56> next_to_clean <8>buffer_info[next_to_clean]: time_stamp <103160197> next_to_watch <9> jiffies <103160cc0> next_to_watch.status <0>MAC Status <40080083>PHY Status <796d>PHY 1000BASE-T Status <3800>PHY Extended Status <3000>PCI Status <10>
Apr 13 21:35:24 vrProxmox kernel: e1000e 0000:00:1f.6 eno2: Detected Hardware Unit Hang: TDH <8> TDT <56> next_to_use <56> next_to_clean <8>buffer_info[next_to_clean]: time_stamp <103160197> next_to_watch <9> jiffies <103161480> next_to_watch.status <0>MAC Status <40080083>PHY Status <796d>PHY 1000BASE-T Status <3800>PHY Extended Status <3000>PCI Status <10>
Apr 13 21:35:26 vrProxmox kernel: e1000e 0000:00:1f.6 eno2: Detected Hardware Unit Hang: TDH <8> TDT <56> next_to_use <56> next_to_clean <8>buffer_info[next_to_clean]: time_stamp <103160197> next_to_watch <9> jiffies <103161c80> next_to_watch.status <0>MAC Status <40080083>PHY Status <796d>PHY 1000BASE-T Status <3800>PHY Extended Status <3000>PCI Status <10>

and it continue like that until you reboot

i am now trying the solution from sudsy from here https://forum.proxmox.com/threads/system-loses-networking.98705/
i also have a VLAN tag from my router, maybe thats the problem.

and added this to the /etc/network/interfaces:

auto vmbr0.40
iface vmbr0.40 inet static
address 192.168.1.4/24
gateway 192.168.1.1

i am still having the same issue, anyone has a solution ?

muzicman0 · Apr 15, 2025

You interfaces file is not correct. Look at mine below yours should be similar. Don't use the command on the bridge, only the NIC. Also, it is a post-up command, you seem to list them all the options as independent lines in your file.

Code:

auto lo
iface lo inet loopback

iface eno2 inet manual
        post-up ethtool -K eno2  gso off tso off rxvlan off txvlan off gro off tx off rx off sg off

auto vmbr0
iface vmbr0 inet static
        address 10.1.10.227/24
        gateway 10.1.10.1
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 100-118 254

source /etc/network/interfaces.d/*

valr30 · Apr 15, 2025

valr30 said:
i am still having the same issue, anyone has a solution ?

ok, i have it fix, it was this solution https://first2host.co.uk/blog/how-to-fix-proxmox-detected-hardware-unit-hang/
and my interfaces looks like this in the end :

auto lo
iface lo inet loopback

iface eno2 inet manual
post-up /sbin/ethtool -K eno2 gso off tso off rxvlan off txvlan off

auto vmbr0
iface vmbr0 inet static
address 192.168.1.4/24
gateway 192.168.1.1
bridge-ports eno2
bridge-stp off
bridge-fd 0
source /etc/network/interfaces.d/*

DimaKompot · Apr 15, 2025

Yesterday I ran an upgrade of the Proxmox host, which I hadn't upgraded since forever. Today my network crashed with the error messages:

Code:

Apr 15 22:09:11 pve kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
  TDH                  <f5>
  TDT                  <2>
  next_to_use          <2>
  next_to_clean        <f4>
buffer_info[next_to_clean]:
  time_stamp           <10c6bcd4b>
  next_to_watch        <f5>
  jiffies              <10c6bd600>
  next_to_watch.status <0>
MAC Status             <40080083>
PHY Status             <796d>
PHY 1000BASE-T Status  <3800>
PHY Extended Status    <3000>
PCI Status             <10>

My ethernet adapter:

Code:

root@pve:~# lspci -v | grep Ethernet
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (6) I219-V (rev 30)
        Subsystem: Intel Corporation Ethernet Connection (6) I219-V

Trying to fix my network with following /etc/network/interfaces:

Code:

root@pve:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual
  post-up /sbin/ethtool -K eno1 tso off gso off rxvlan off txvlan off

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.115/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface wlp0s20f3 inet manual


source /etc/network/interfaces.d/*

Too bad I cannot just restore the backup of host, like I can do it with VMs

[SOLVED] Intel NIC e1000e hardware unit hang

New Member

New Member

New Member

Attachments

New Member

New Member

Renowned Member

New Member

New Member

New Member

Member

Member

Member

Member

Member

New Member

Attachments

New Member

New Member

Member

New Member

New Member

We value your privacy