Detected Hardware Unit Hang

[niekniek89]

Hello,

I can't find the right answer to my question right away, so I'm opening a new post.
I've had my Proxmox host suddenly become unreachable several times now.
The log shows the following message:

Dec 31 19:55:58 Proxmox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang

I've already searched the internet for some things, but I can't find the right one.
I also saw this online:

post-up /sbin/ethtool -K eno1 tso off gso off gro off

But before I run that, I see that everything is already set to "off" for me.
I've never made such a change to the NIC before.

root@Proxmox01:~# ethtool -k eno1 | grep -E 'tso|gso|gro'
tx-gso-robust: off [fixed]
tx-gso-partial: off [fixed]
tx-gso-list: off [fixed]
rx-gro-hw: off [fixed]
rx-gro-list: off
rx-udp-gro-forwarding: off
root@Proxmox01:~#

I do have proxmox in a separate mgmt vlan:

iface vmbr0 inet manual
bridge ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge vids 2-4094

iface wlp0s20f3 inet manual

auto vmbr0.3
iface vmbr0.3 inet static
address 192.168.176.10/24
gateway 192.168.176.1

This has always worked without any problems, but since v9.x, I've noticed this issue where it frequently drops out.
Disconnecting and reconnecting the network cable fixes this temporarily.

Does anyone have any ideas how to fix this?
Thanks in advance for your help!

the output from nano /etc/network/interfaces

 

[SneedsFeedAndSeed]

This is a common problem for these nics. While yours should be fine with tso/gro/gso off, often the recommended solution is do disable all offloading features. Probably overkill, but it's worth a shot.

ethtool -K eno1 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off

And you can turn this into a systemd service to run once on boot using something like this:

#/etc/systemd/system/e1000e-offload-disable.service
[Unit]
Description=Disable NIC offloading for Intel e1000e interface eno1
After=network.target

[Service]
Type=oneshot
ExecStart=/sbin/ethtool -K eno1 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

Then enable/start the service:

systemctl enable --now e1000e-offload-disable.service

 

[niekniek89]

Thanks for your help and quick response!
What's the easiest/fastest thing to do right now?

Make the change in "nano /etc/network/interfaces"?

 

[SneedsFeedAndSeed]

You won't make any changes to /etc/network/interfaces, that file looks file.

The quickest (possible) solution would be running:

ethtool -K eno1 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off

Note: this is not persistent. To make persistent, create the systemd service file I poasted above. There is also a helper script that does this.

 

[niekniek89]

is a host reboot required after running the script?

 

[SneedsFeedAndSeed]

No

 

[niekniek89]

Thanks for your help.
i installed the proxmox script succesfully;

Enhanced version supporting both e1000e and e1000 drivers

Searching for Intel e1000e and e1000 interfaces...
Found 1 Intel e1000e/e1000 interfaces
Selected interface: eno1 (e1000e)
Creating systemd service for interface: eno1 (e1000e)...
Service for eno1 (e1000e) created and enabled!
Service: disable-nic-offload-eno1.service...
Status: Active...
Start on boot: Enabled...
Intel e1000e/e1000 optimization complete for 1 interface(s)!

Verification commands:...
ethtool -k eno1 # Check offloading status
systemctl status disable-nic-offload-eno1.service # Check service status

=========================================================================================

root@Proxmox01:~# ethtool -k eno1
Features for eno1:
rx-checksumming: off
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
tx-tcp-accecn-segmentation: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
root@Proxmox01:~#