e1000e network issue on Proxmox

benj29

Member
Nov 6, 2019
8
2
23
49
Hi,

Happy new year :) !!

Yesterday, i've lost the ethernet connection on my Proxmox.
Console is okay, but no IP available.
A restart of the server will make available the server on the network.
My server is a NUC based on DeskMini 310 Series. After checking the doc, the LAN is based on Intel® Gigabit I219V.

After checking the logs, repetitively every 5 minutes, due to the lack of network interfaces until the morning... :

Code:
Jan 02 21:35:09 pve kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
                              TDH                  <0>
                              TDT                  <1f>
                              next_to_use          <1f>
                              next_to_clean        <0>
                            buffer_info[next_to_clean]:
                              time_stamp           <14eb7499f>
                              next_to_watch        <0>
                              jiffies              <14eb752f8>
                              next_to_watch.status <0>
                            MAC Status             <40080083>
                            PHY Status             <796d>
                            PHY 1000BASE-T Status  <3c00>
                            PHY Extended Status    <3000>
                            PCI Status             <10>
Jan 02 21:35:09 pve kernel: e1000e 0000:00:1f.6 eno1: Reset adapter unexpectedly

- mypve :
Code:
proxmox-ve: 8.0.2 (running kernel: 6.2.16-15-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
proxmox-kernel-6.2: 6.2.16-15
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.26-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.9
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.3-1
proxmox-backup-file-restore: 3.0.3-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.9
pve-cluster: 8.0.4
pve-container: 5.0.4
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.13-pve1

- my NIC:
Code:
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
        DeviceName: Onboard - Ethernet
        Subsystem: ASRock Incorporation Ethernet Connection (7) I219-V [1849:15bc]
        Kernel driver in use: e1000e
        Kernel modules: e1000e

- my interfaces :
Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.2.80/24
        gateway 192.168.2.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

- my ethtool summary :
Code:
Features for eno1:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

After some research, seems to be a bug on this cheap network card.
https://forum.proxmox.com/threads/e1000-driver-hang.58284/page-8#post-390709

But tso, gro, gso are already OFF in my configuration.
Any tips to modify the interfaces configuration to reach an ideal ethtool configuration of my eno1 ?

Thank you !
 
Last edited:
  • Like
Reactions: Nik0
Hi,

Happy new year :) !!

Yesterday, i've lost the ethernet connection on my Proxmox.
Console is okay, but no IP available.
A restart of the server will make available the server on the network.
My server is a NUC based on DeskMini 310 Series. After checking the doc, the LAN is based on Intel® Gigabit I219V.

After checking the logs, repetitively every 5 minutes, due to the lack of network interfaces until the morning... :

Code:
Jan 02 21:35:09 pve kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
                              TDH                  <0>
                              TDT                  <1f>
                              next_to_use          <1f>
                              next_to_clean        <0>
                            buffer_info[next_to_clean]:
                              time_stamp           <14eb7499f>
                              next_to_watch        <0>
                              jiffies              <14eb752f8>
                              next_to_watch.status <0>
                            MAC Status             <40080083>
                            PHY Status             <796d>
                            PHY 1000BASE-T Status  <3c00>
                            PHY Extended Status    <3000>
                            PCI Status             <10>
Jan 02 21:35:09 pve kernel: e1000e 0000:00:1f.6 eno1: Reset adapter unexpectedly

- mypve :
Code:
proxmox-ve: 8.0.2 (running kernel: 6.2.16-15-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
proxmox-kernel-6.2: 6.2.16-15
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.26-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.9
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.3-1
proxmox-backup-file-restore: 3.0.3-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.9
pve-cluster: 8.0.4
pve-container: 5.0.4
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.13-pve1

- my NIC:
Code:
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
        DeviceName: Onboard - Ethernet
        Subsystem: ASRock Incorporation Ethernet Connection (7) I219-V [1849:15bc]
        Kernel driver in use: e1000e
        Kernel modules: e1000e

- my interfaces :
Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.2.80/24
        gateway 192.168.2.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

- my ethtool summary :
Code:
Features for eno1:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

After some research, seems to be a bug on this cheap network card.
https://forum.proxmox.com/threads/e1000-driver-hang.58284/page-8#post-390709

But tso, gro, gso are already OFF in my configuration.
Any tips to modify the interfaces configuration to reach an ideal ethtool configuration of my eno1 ?

Thank you !

I found out today that I have the exact same issue using the deskmini h310
You mention that
But tso, gro, gso are already OFF in my configuration.
However the post you point to states that you should disable ALL offloading because in his case only disabling tso or gso didn't help.
Your config seems to still have (some) offloading enabled

Code:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ip-generic: on
generic-segmentation-offload: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on

Have you tried using the full command the poster stated:
Bash:
ethtool -K eth0 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off
And then comparing the ethtool -k output?

I ran the above command and now we just have to wait and see if the error returns..in my case it can be found using the command:
Bash:
journalctl | grep Hang
Dec 12 12:11:42 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 12 12:11:44 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 12 12:11:46 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 12 12:11:48 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 12 12:11:50 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 17 08:32:44 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 17 08:32:46 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 17 08:32:48 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 30 23:56:39 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 30 23:56:41 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 30 23:56:43 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 30 23:56:45 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Dec 30 23:56:47 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:48:55 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:48:57 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:48:59 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:49:01 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:49:03 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:50:45 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:50:47 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:50:49 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:56:24 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:56:26 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:56:28 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 16:56:30 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 17:06:13 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 17:06:15 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 17:06:17 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 17:06:32 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 17:06:34 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang: Jan 07 17:06:36 prox01 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
 
Hi,

I have not updated the thread, but i modify my configuration and it seems to be OK since 1 week.

ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096

Do you recommand me to off other settings ? but since 1 week, no new crash....
 
Hi,

I have not updated the thread, but i modify my configuration and it seems to be OK since 1 week.

ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096

Do you recommand me to off other settings ? but since 1 week, no new crash....
I don't know what prompted you to set those specific settings?
 
At this time, the -k command reports :

Code:
Features for eno1:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ip-generic: on
scatter-gather: on
        tx-scatter-gather: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
receive-hashing: on

So, samething than my last reports.

The journalctl | grep Hang reports nothing since 3rd of January, BEFORE the changings.
Maybe... this is why I say it is good now :). I've tried to transfer a lot of data (> 1 Tb per day) and it is worked without errors on the journal.

I've found this recommandations in the forum with the kind of errors :
https://forum.proxmox.com/threads/e1000-driver-hang.58284/page-10
 
  • Like
Reactions: Nik0
At this time, the -k command reports :

Code:
Features for eno1:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ip-generic: on
scatter-gather: on
        tx-scatter-gather: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
receive-hashing: on

So, samething than my last reports.

The journalctl | grep Hang reports nothing since 3rd of January, BEFORE the changings.
Maybe... this is why I say it is good now :). I've tried to transfer a lot of data (> 1 Tb per day) and it is worked without errors on the journal.

I've found this recommandations in the forum with the kind of errors :
https://forum.proxmox.com/threads/e1000-driver-hang.58284/page-10
If it works I wouldn't touch it any further :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!