e1000e:reset adapter unexpectedly

L

lxiosjao

Member
Mar 21, 2021
63
1
13
Hello,

I have the latest Proxmox installed with an Intel 82579V Gb network controller.
Proxmox ui (and CT/VM) are not reachable for a few seconds.
This would happen up to several times a day at seemingly random.
What am I supposed to do to solve that issue ?
Is it a driver or hardware issue ?

screen error.png
 
Last edited:
Hi,

lxiosjao said:
Is it a driver or hardware issue ?
Click to expand...
There a high chance that this is the HW/Firmware issue which certain models using the e1000e driver have, basically receive/transmit offload is somewhat broken there. The Linux kernel and the e1000e module it ships cannot do anything about that.

If it is really that issue, which is likely, then you can disable the receive and send offload functions as workaround:

Bash: 
apt install ethtool
ethtool -K $eno1 tso off gso off gro off

This is for the current boot only.

To apply it automatically on boot edit /etc/network/interfaces and add a post-up entry for that interface (can also add it on the bridge):

Code: 
iface eno1 inet manual
    # ... existing lines
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off
 
  • Like
Reactions: lxiosjao, Eleferen, Tatariushi and 2 others
Hi,
Thanks for this quick answer.
I am going to try your workaround.
What does it do exactly?
 
lxiosjao said:
What does it do exactly?
Click to expand...
t.lamprecht said:
then you can disable the receive and send offload functions as workaround:
Click to expand...

As said, it disables some off-loading functions (basically some type of hardware acceleration for sending/receiving packets) of the NIC, that means the host CPU needs to do slightly more work, for the benefit of a working NIC.
 
  • Like
Reactions: lxiosjao and Deepen Dhulla
t.lamprecht said:
To apply it automatically on boot edit /etc/network/interfaces and add a post-up entry for that interface (can also add it on the bridge):

Code: 
iface eno1 inet manual
    # ... existing lines
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off
Click to expand...

Thomas, how is the $IFACE variable expanded? There's a post-up script that processes that line and has it defined?

Thanks.
 
No need to replace it, it's an env variable that contains the interface for the scripts in /etc/network/if-*.d

Every time an interface comes up or goes down, scripts in those dirs are executed and that variable is set with the inteface name.

So now I'm using this, and it works perfectly:

Code: 
auto lo
iface lo inet loopback


auto eno1
iface eno1 inet static
    address 10.1.10.12/23
    gateway 10.1.10.1
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off
 
Last edited:
Ah, good to know, wasn't aware that this was a universal variable. Very cool. Thank you.
 
Hello, I'm having the same issue. Tried the proposed workaround (disabling off-loading) but the issue is still present.
We have a 3 node cluster over Dell R740xd servers. Recently we update from PVE 6.4 to 7.4-16. No changes.
I'm wondering if changing to VirtIO can solve the problem???

Code: 
# zcat syslog.6.gz | grep -ie kernel -ie keepalived | grep -v Cannot
Aug 18 17:07:20 host kernel: [23593194.904087] e1000 0000:00:15.0 eth1: Reset adapter
Aug 18 17:07:22 host kernel: [23593196.984494] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

Code: 
# pveversion --verbose
proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-4
pve-kernel-5.4: 6.4-20
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 16.2.13-pve1
ceph-fuse: 16.2.13-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.3-1
proxmox-backup-file-restore: 2.4.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-5
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

Code: 
# cat /etc/pve/qemu-server/1034.conf 
agent: 1
bootdisk: scsi0
cores: 12
cpu: kvm64,flags=+aes
hotplug: disk,network,usb,cpu
ide2: none,media=cdrom
memory: 2048
name: host
net0: e1000=3E:20:FD:8F:64:5A,bridge=vmbr0,firewall=1,tag=92
net1: e1000=FA:96:D8:E8:9A:0A,bridge=vmbr0,firewall=1,tag=910
net2: e1000=02:63:87:0B:E5:B3,bridge=vmbr0,firewall=1,tag=932
net3: e1000=02:0C:05:5D:7F:7B,bridge=vmbr0,firewall=1,tag=337
numa: 0
onboot: 1
ostype: l26
scsi0: pool:vm-1034-disk-0,discard=on,size=20G
scsihw: virtio-scsi-pci
smbios1: uuid=a9d66c4e-8afd-4af2-b597-50e2450047e8
sockets: 1
vcpus: 6
vmgenid: 6826e577-b999-4c32-aa88-9f93f13d86b5
Code: 
# dmesg | grep -i ethernet
[    2.411983] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[    2.411989] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.770810] igb 0000:19:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.282897] igb 0000:19:00.1: Intel(R) Gigabit Ethernet Network Connection
[    3.676975] igb 0000:19:00.2: Intel(R) Gigabit Ethernet Network Connection
[    3.746888] igb 0000:19:00.3: Intel(R) Gigabit Ethernet Network Connection

Code: 
# ethtool -i eno1
driver: igb
version: 5.15.108-1-pve
firmware-version: 1.67, 0x80000faa, 19.5.12
expansion-rom-version: 
bus-info: 0000:19:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Code: 
# ethtool -k eno1
Features for eno1:
rx-checksumming: on
tx-checksumming: on
    tx-checksum-ipv4: off [fixed]
    tx-checksum-ip-generic: on
    tx-checksum-ipv6: off [fixed]
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: on
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
    tx-tcp-segmentation: off
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp-mangleid-segmentation: off
    tx-tcp6-segmentation: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
 
alexdelprete said:
No need to replace it, it's an env variable that contains the interface for the scripts in /etc/network/if-*.d

Every time an interface comes up or goes down, scripts in those dirs are executed and that variable is set with the inteface name.

So now I'm using this, and it works perfectly:

Code: 
auto lo
iface lo inet loopback


auto eno1
iface eno1 inet static
    address 10.1.10.12/23
    gateway 10.1.10.1
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off
Click to expand...
Should I add this to my physical interface or my bridge?

Code: 
cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual
        post-up /sbin/ethtool -K $IFACE tso off gso off gro off

auto vmbr0
iface vmbr0 inet static
        address 10.0.1.10/24
        gateway 10.0.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
 
dustojnikhummer said:
Should I add this to my physical interface or my bridge?
Click to expand...

On the bridge:

Bash: 
> cat /etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 10.1.10.10/23
    gateway 10.1.10.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off rxvlan off txvlan off
 
Thanks. And just to be sure, this only affects the Intel e1000e driver, for Intel NICs? I have two very similar HP miniPCs, but the second one has a Realtek NIC. I assume that one doesn't need this "fix"?
 
dustojnikhummer said:
Thanks. And just to be sure, this only affects the Intel e1000e driver, for Intel NICs? I have two very similar HP miniPCs, but the second one has a Realtek NIC. I assume that one doesn't need this "fix"?
Click to expand...

from what I researched at the time, the issue is specific with e1000e driver and it came up only because my PBS backups were failing with strange communication errors. If you don't have any issues with those HP mini-pcs, don't bother.
 
sdet00 said:
Any idea if this is still a problem in Proxmox 8?
Click to expand...
I am wanting to know this also.

I recently nuked and rebuilt my home lab PVE with the latest download, and it went from rock solid on 7.1.2 (or somewhere near that version) to dropping my network interface every 30 seconds.

i have just inserted it into my config like below. Will monitor for results.

Code: 
root@pve:/var/log# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual
post-up /sbin/ethtool -K $IFACE tso off gso off gro off


auto vmbr0
iface vmbr0 inet static
        address 192.168.100.25/24
        gateway 192.168.100.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface enp8s0 inet manual
 
jbates58 said:
I am wanting to know this also.

I recently nuked and rebuilt my home lab PVE with the latest download, and it went from rock solid on 7.1.2 (or somewhere near that version) to dropping my network interface every 30 seconds.

i have just inserted it into my config like below. Will monitor for results.

Code: 
root@pve:/var/log# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual
post-up /sbin/ethtool -K $IFACE tso off gso off gro off


auto vmbr0
iface vmbr0 inet static
        address 192.168.100.25/24
        gateway 192.168.100.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface enp8s0 inet manual
Click to expand...
I was told (three messages above this) you are supposed to put it on the bridge, not the physical interface
 
yes, i saw that. I currently have it on the interface, and after many reboots it seems to be working properly for the moment.
im trying to get my melanox CX3 passed through to a Truenas VM, and am not having much luck. so reboots are a plenty haha
 
jbates58 said:
yes, i saw that. I currently have it on the interface, and after many reboots it seems to be working properly for the moment.
im trying to get my melanox CX3 passed through to a Truenas VM, and am not having much luck. so reboots are a plenty haha
Click to expand...

How is it going, any more issues to report?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back