e1000 driver hang
- Thread starter alatteri
- Start date
After two months of dealing with crashes and being unable to solve the problem on some 1L PCs, I tried using a few realtek USB gigabit network adapters. It seems the hosts are still crashing, despite the onboard intel NIC being fully disabled.
Will look more tomorrow. Tried the solutions outlined by others, but haven't found anything that works for me yet.
Will look more tomorrow. Tried the solutions outlined by others, but haven't found anything that works for me yet.
Such a weird error, my home server worked fine for years, and since i switched to a new ISP the problems suddenly starts.. (from cable to Fiber)
if the offloading fix does not work, i just put in a pci-e x1 card i think. Would an Intel 82574L Chipset be good enough and not vunerable for this glitch?
if the offloading fix does not work, i just put in a pci-e x1 card i think. Would an Intel 82574L Chipset be good enough and not vunerable for this glitch?
Same problem here, at first I've tried to disable rx/tx vlan offload but nothing changed, now I'll try with the two/gso and I'll tell you if it works.
I'm on proxmox VE 7.2 on Intel NUC 10th Gen:
Code:
# lspci -v
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (10) I219-V
DeviceName: LAN
Subsystem: Intel Corporation Ethernet Connection (10) I219-V
Flags: bus master, fast devsel, latency 0, IRQ 145, IOMMU group 11
Memory at 56300000 (32-bit, non-prefetchable) [size=128K]
Capabilities: [c8] Power Management version 3
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Kernel driver in use: e1000e
Kernel modules: e1000e
# uname -a
Linux pve 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64 GNU/LinuxUPDATE: it works for me, issue disappeared disabling only gso and tso
Last edited:
to confirm, to fix this, you added the below code to /etc/network/interfaces?
Code:
iface eno1 inet manual
offload-tso off
offload-gso off
Last edited:
No, I didn’t try that way and I directly used the post-up command
Code:
iface eno1 inet manual
post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"
Last edited:
Thanks. I've done that and rebooted the host. It did not fix it for me. I've attached logging and the interfaces(.txt) file.
How can I verify TSO and GSO are succesfully disabled?
Someone said that TSO and GSO were not sufficient and disabled also GRO.
Here you can see if segmentation parameters are off:
Bash:
# ethtool -k eno1 | grep -i segmentation
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
generic-segmentation-offload: off
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]For example you can try disabling vlan rx/tx offload or GRO too and see if it helps. On my setup the following are still on but give no issues:
Bash:
# ethtool -k eno1 | grep -i ": on"
rx-checksumming: on
tx-checksumming: on
tx-checksum-ip-generic: on
scatter-gather: on
tx-scatter-gather: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
receive-hashing: on
highdma: on [fixed]GRO is the
generic-receive-offload above.You can try also disabling parameters interactively, in order to see if you're allowed to disable them and then use the command
ethtool -k eno1 to see if they've been correctly disabled. If it works and the problem disappear, update the command in the /etc/network/interfaces post-up section.To disable rx-vlan-offload for example you can run
ethtool -K eno1 rxvlan off. The parameters are named with shortnames that you find in the man page:
Code:
-K --features --offload
Changes the offload parameters and other features of the specified network device. The following feature names are built-in and others may be defined by the kernel.
rx on|off
Specifies whether RX checksumming should be enabled.
tx on|off
Specifies whether TX checksumming should be enabled.
sg on|off
Specifies whether scatter-gather should be enabled.
tso on|off
Specifies whether TCP segmentation offload should be enabled.
ufo on|off
Specifies whether UDP fragmentation offload should be enabled
gso on|off
Specifies whether generic segmentation offload should be enabled
gro on|off
Specifies whether generic receive offload should be enabled
lro on|off
Specifies whether large receive offload should be enabled
rxvlan on|off
Specifies whether RX VLAN acceleration should be enabled
txvlan on|off
Specifies whether TX VLAN acceleration should be enabled
ntuple on|off
Specifies whether Rx ntuple filters and actions should be enabled
rxhash on|off
Specifies whether receive hashing offload should be enabled
Last edited:
Thanks for the instructions. I'll make changes and report back
Quite frustrating that this problem is still ongoing today... I had this issue around a year or two back, and put together a new Proxmox system (well, it's refurbished) and forgot about this bug. Workarounds did the trick thankfully.
stable but with a significant drop in performance with
iface eno1 inet manual
pre-up /usr/sbin/ethtool -K eno1 tso off gso off
Anyone able to get the "e1000e 0000:00:1f.6 eno1: Detected Hardware Unit" worked around w/o performance degradation ?
iface eno1 inet manual
pre-up /usr/sbin/ethtool -K eno1 tso off gso off
Anyone able to get the "e1000e 0000:00:1f.6 eno1: Detected Hardware Unit" worked around w/o performance degradation ?
What kind of degradation do you experience? I have those two parameters disabled and I’ve tested about 930Mbps on the 1Gbps e1000 interface
Code:
grep post /etc/network/interfaces
post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"
Code:
iperf3 -c 10.0.0.3 -i 10 -p 5201 -t 10s -w 416k -P 4
Connecting to host 10.0.0.3, port 5201
[ 5] local 10.0.0.4 port 43410 connected to 10.0.0.3 port 5201
[ 7] local 10.0.0.4 port 43424 connected to 10.0.0.3 port 5201
[ 9] local 10.0.0.4 port 43434 connected to 10.0.0.3 port 5201
[ 11] local 10.0.0.4 port 43444 connected to 10.0.0.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-10.00 sec 285 MBytes 239 Mbits/sec 0 458 KBytes
[ 7] 0.00-10.00 sec 276 MBytes 231 Mbits/sec 0 451 KBytes
[ 9] 0.00-10.00 sec 276 MBytes 232 Mbits/sec 0 452 KBytes
[ 11] 0.00-10.00 sec 276 MBytes 231 Mbits/sec 0 450 KBytes
[SUM] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 285 MBytes 239 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 285 MBytes 239 Mbits/sec receiver
[ 7] 0.00-10.00 sec 276 MBytes 231 Mbits/sec 0 sender
[ 7] 0.00-10.00 sec 275 MBytes 231 Mbits/sec receiver
[ 9] 0.00-10.00 sec 276 MBytes 232 Mbits/sec 0 sender
[ 9] 0.00-10.00 sec 276 MBytes 231 Mbits/sec receiver
[ 11] 0.00-10.00 sec 276 MBytes 231 Mbits/sec 0 sender
[ 11] 0.00-10.00 sec 276 MBytes 231 Mbits/sec receiver
[SUM] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec 0 sender
[SUM] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec receiver
Last edited:
I'm also getting 930Mbps - moving on to debug the router/ISP settings.
I appreciate the help
Thank you
930Mbps is about the maximum possible throughput with TCP, so there seems to be no performance impact due to disabling TSO and GSO offload.
You could reach 980Mbps with jumbo frames.
What is the actual maximum throughput on Gigabit Ethernet?
I always have to reboot my machine twice because of this. None of the fixes seems to work 100% for me. Like today, my PVE was online for 38 days. I've ran my updates and everytime again, 5 minutes after a reboot, my pve freezes. Hard power off + power on reboots it and it is able to stay online for weeks/months/... Only the first warm reboot after updates freezes after 5 minutes every time again.
So my procedure is:
> Running for weeks/months
> Update PVE
> Reboot node
> FREEZES after 5 minutes
> Hard power off + power on (using a Sonoff now because I'm lazy)
> Boots up autom. because of boot on power setting
> Running for weeks/months
So my procedure is:
> Running for weeks/months
> Update PVE
> Reboot node
> FREEZES after 5 minutes
> Hard power off + power on (using a Sonoff now because I'm lazy)
> Boots up autom. because of boot on power setting
> Running for weeks/months
Last edited:
my current
/etc/network/interfaces file looks like this
Code:
auto lo
iface lo inet loopback
iface eno1 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.61/24
gateway 192.168.1.1
bridge-ports eno1
bridge-stp off
bridge-fd 0eno1 or vmbr0? I'm a bit confused since the bridge one has the ip. Thanks.Add it under the physical interface eno1. I have this under eno1:
Code:
iface eno1 inet manual
post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"
Last edited:
Hi, the ethtool fix works fine, and I've set-up a post-up action to run in each time eno1 is up.
But after adding a new network interface on one of my vm with the VLAN tag 20, the host freeze issue returns.
This is my current interfaces file:
But apparently, ProxmoxVE creates a new linux interface dynamically to add/manage the VLAN ... and ethtool fixes must also be applied on it.
Currently I'm running ethtool manually on the eno1.20 VLAN interface after each restart (and that is fixing the host freeze issue again), but I would like it to be done automatically like for eno1.
So what is the proper way to add ethtool post-up to interfaces for eno1.20 ?
Is there will be conflicts with the fact that Proxmox already manage it internally ?
But after adding a new network interface on one of my vm with the VLAN tag 20, the host freeze issue returns.
This is my current interfaces file:
Bash:
auto lo
iface lo inet loopback
iface eno1 inet manual
post-up /sbin/ethtool -K $IFACE tso off gso off gro off
auto vmbr0
iface vmbr0 inet static
address 192.168.10.3/24
gateway 192.168.10.1
bridge-ports eno1
bridge-stp off
bridge-fd 0But apparently, ProxmoxVE creates a new linux interface dynamically to add/manage the VLAN ... and ethtool fixes must also be applied on it.
Bash:
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 2c:f0:5d:db:ce:81 txqueuelen 1000 (Ethernet)
RX packets 45641515 bytes 56862382902 (52.9 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23033002 bytes 14289197449 (13.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xa4500000-a4520000
eno1.20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 2c:f0:5d:db:ce:81 txqueuelen 1000 (Ethernet)
RX packets 1788798 bytes 218498559 (208.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3137555 bytes 4349237578 (4.0 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
... tapxxx/fwxx interfaces removed
vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.3 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::2ef0:5dff:fedb:ce81 prefixlen 64 scopeid 0x20<link>
ether 2c:f0:5d:db:ce:81 txqueuelen 1000 (Ethernet)
RX packets 38069900 bytes 55406252647 (51.6 GiB)
RX errors 0 dropped 215192 overruns 0 frame 0
TX packets 4793735 bytes 464390058 (442.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vmbr0v20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 9e:e3:f7:28:36:2f txqueuelen 1000 (Ethernet)
RX packets 77057 bytes 6628091 (6.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0Currently I'm running ethtool manually on the eno1.20 VLAN interface after each restart (and that is fixing the host freeze issue again), but I would like it to be done automatically like for eno1.
So what is the proper way to add ethtool post-up to interfaces for eno1.20 ?
Is there will be conflicts with the fact that Proxmox already manage it internally ?