e1000e:reset adapter unexpectedly

[lxiosjao]

Hello,

I have the latest Proxmox installed with an Intel 82579V Gb network controller.
Proxmox ui (and CT/VM) are not reachable for a few seconds.
This would happen up to several times a day at seemingly random.
What am I supposed to do to solve that issue ?
Is it a driver or hardware issue ?

 

[t.lamprecht]

Hi,

Is it a driver or hardware issue ?

There a high chance that this is the HW/Firmware issue which certain models using the e1000e driver have, basically receive/transmit offload is somewhat broken there. The Linux kernel and the e1000e module it ships cannot do anything about that.

If it is really that issue, which is likely, then you can disable the receive and send offload functions as workaround:

apt install ethtool
ethtool -K $eno1 tso off gso off gro off

This is for the current boot only.

To apply it automatically on boot edit /etc/network/interfaces and add a post-up entry for that interface (can also add it on the bridge):

iface eno1 inet manual
    # ... existing lines
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

 

[lxiosjao]

Hi,
Thanks for this quick answer.
I am going to try your workaround.
What does it do exactly?

 

[t.lamprecht]

What does it do exactly?

then you can disable the receive and send offload functions as workaround:

As said, it disables some off-loading functions (basically some type of hardware acceleration for sending/receiving packets) of the NIC, that means the host CPU needs to do slightly more work, for the benefit of a working NIC.

 

[lxiosjao]

Thank you very much.

 

[alexdelprete]

To apply it automatically on boot edit /etc/network/interfaces and add a post-up entry for that interface (can also add it on the bridge):

iface eno1 inet manual
    # ... existing lines
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

Thomas, how is the $IFACE variable expanded? There's a post-up script that processes that line and has it defined?

Thanks.

 

[sdet00]

Thomas, how is the $IFACE variable expanded? There's a post-up script that processes that line and has it defined?

Not quire sure what you're asking for here - I just replaced $IFACE with my interface ID, in my case, eno1.

Screenshot that might help: https://i.imgur.com/7POUGnu.jpeg

 

[alexdelprete]

No need to replace it, it's an env variable that contains the interface for the scripts in /etc/network/if-*.d

Every time an interface comes up or goes down, scripts in those dirs are executed and that variable is set with the inteface name.

So now I'm using this, and it works perfectly:

auto lo
iface lo inet loopback


auto eno1
iface eno1 inet static
    address 10.1.10.12/23
    gateway 10.1.10.1
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

 

[sdet00]

Ah, good to know, wasn't aware that this was a universal variable. Very cool. Thank you.

 

[Nicolas Lobariñas]

Hello, I'm having the same issue. Tried the proposed workaround (disabling off-loading) but the issue is still present.
We have a 3 node cluster over Dell R740xd servers. Recently we update from PVE 6.4 to 7.4-16. No changes.
I'm wondering if changing to VirtIO can solve the problem???

# zcat syslog.6.gz | grep -ie kernel -ie keepalived | grep -v Cannot
Aug 18 17:07:20 host kernel: [23593194.904087] e1000 0000:00:15.0 eth1: Reset adapter
Aug 18 17:07:22 host kernel: [23593196.984494] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

# pveversion --verbose
proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-4
pve-kernel-5.4: 6.4-20
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 16.2.13-pve1
ceph-fuse: 16.2.13-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.3-1
proxmox-backup-file-restore: 2.4.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-5
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

# cat /etc/pve/qemu-server/1034.conf 
agent: 1
bootdisk: scsi0
cores: 12
cpu: kvm64,flags=+aes
hotplug: disk,network,usb,cpu
ide2: none,media=cdrom
memory: 2048
name: host
net0: e1000=3E:20:FD:8F:64:5A,bridge=vmbr0,firewall=1,tag=92
net1: e1000=FA:96:D8:E8:9A:0A,bridge=vmbr0,firewall=1,tag=910
net2: e1000=02:63:87:0B:E5:B3,bridge=vmbr0,firewall=1,tag=932
net3: e1000=02:0C:05:5D:7F:7B,bridge=vmbr0,firewall=1,tag=337
numa: 0
onboot: 1
ostype: l26
scsi0: pool:vm-1034-disk-0,discard=on,size=20G
scsihw: virtio-scsi-pci
smbios1: uuid=a9d66c4e-8afd-4af2-b597-50e2450047e8
sockets: 1
vcpus: 6
vmgenid: 6826e577-b999-4c32-aa88-9f93f13d86b5

# dmesg | grep -i ethernet
[    2.411983] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[    2.411989] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.770810] igb 0000:19:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.282897] igb 0000:19:00.1: Intel(R) Gigabit Ethernet Network Connection
[    3.676975] igb 0000:19:00.2: Intel(R) Gigabit Ethernet Network Connection
[    3.746888] igb 0000:19:00.3: Intel(R) Gigabit Ethernet Network Connection

# ethtool -i eno1
driver: igb
version: 5.15.108-1-pve
firmware-version: 1.67, 0x80000faa, 19.5.12
expansion-rom-version: 
bus-info: 0000:19:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -k eno1
Features for eno1:
rx-checksumming: on
tx-checksumming: on
    tx-checksum-ipv4: off [fixed]
    tx-checksum-ip-generic: on
    tx-checksum-ipv6: off [fixed]
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: on
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
    tx-tcp-segmentation: off
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp-mangleid-segmentation: off
    tx-tcp6-segmentation: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

 

[sdet00]

Any idea if this is still a problem in Proxmox 8?

 

[dustojnikhummer]

No need to replace it, it's an env variable that contains the interface for the scripts in /etc/network/if-*.d

Every time an interface comes up or goes down, scripts in those dirs are executed and that variable is set with the inteface name.

So now I'm using this, and it works perfectly:

auto lo
iface lo inet loopback


auto eno1
iface eno1 inet static
    address 10.1.10.12/23
    gateway 10.1.10.1
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

Should I add this to my physical interface or my bridge?

cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual
        post-up /sbin/ethtool -K $IFACE tso off gso off gro off

auto vmbr0
iface vmbr0 inet static
        address 10.0.1.10/24
        gateway 10.0.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

 

[alexdelprete]

Should I add this to my physical interface or my bridge?

On the bridge:

> cat /etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 10.1.10.10/23
    gateway 10.1.10.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off rxvlan off txvlan off

 

[dustojnikhummer]

Thanks. And just to be sure, this only affects the Intel e1000e driver, for Intel NICs? I have two very similar HP miniPCs, but the second one has a Realtek NIC. I assume that one doesn't need this "fix"?

 

[alexdelprete]

Thanks. And just to be sure, this only affects the Intel e1000e driver, for Intel NICs? I have two very similar HP miniPCs, but the second one has a Realtek NIC. I assume that one doesn't need this "fix"?

from what I researched at the time, the issue is specific with e1000e driver and it came up only because my PBS backups were failing with strange communication errors. If you don't have any issues with those HP mini-pcs, don't bother.

 

[jbates58]

Any idea if this is still a problem in Proxmox 8?

I am wanting to know this also.

I recently nuked and rebuilt my home lab PVE with the latest download, and it went from rock solid on 7.1.2 (or somewhere near that version) to dropping my network interface every 30 seconds.

i have just inserted it into my config like below. Will monitor for results.

root@pve:/var/log# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual
post-up /sbin/ethtool -K $IFACE tso off gso off gro off


auto vmbr0
iface vmbr0 inet static
        address 192.168.100.25/24
        gateway 192.168.100.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface enp8s0 inet manual

 

[dustojnikhummer]

I am wanting to know this also.

I recently nuked and rebuilt my home lab PVE with the latest download, and it went from rock solid on 7.1.2 (or somewhere near that version) to dropping my network interface every 30 seconds.

i have just inserted it into my config like below. Will monitor for results.

root@pve:/var/log# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual
post-up /sbin/ethtool -K $IFACE tso off gso off gro off


auto vmbr0
iface vmbr0 inet static
        address 192.168.100.25/24
        gateway 192.168.100.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface enp8s0 inet manual

I was told (three messages above this) you are supposed to put it on the bridge, not the physical interface

 

[jbates58]

yes, i saw that. I currently have it on the interface, and after many reboots it seems to be working properly for the moment.
im trying to get my melanox CX3 passed through to a Truenas VM, and am not having much luck. so reboots are a plenty haha

 

[sdet00]

yes, i saw that. I currently have it on the interface, and after many reboots it seems to be working properly for the moment.
im trying to get my melanox CX3 passed through to a Truenas VM, and am not having much luck. so reboots are a plenty haha

How is it going, any more issues to report?