e1000e:reset adapter unexpectedly

[lxiosjao]

Hello,

I have the latest Proxmox installed with an Intel 82579V Gb network controller.
Proxmox ui (and CT/VM) are not reachable for a few seconds.
This would happen up to several times a day at seemingly random.
What am I supposed to do to solve that issue ?
Is it a driver or hardware issue ?

 

[t.lamprecht]

Hi,

Is it a driver or hardware issue ?

There a high chance that this is the HW/Firmware issue which certain models using the e1000e driver have, basically receive/transmit offload is somewhat broken there. The Linux kernel and the e1000e module it ships cannot do anything about that.

If it is really that issue, which is likely, then you can disable the receive and send offload functions as workaround:

apt install ethtool
ethtool -K $eno1 tso off gso off gro off

This is for the current boot only.

To apply it automatically on boot edit /etc/network/interfaces and add a post-up entry for that interface (can also add it on the bridge):

iface eno1 inet manual
    # ... existing lines
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

 

[lxiosjao]

Hi,
Thanks for this quick answer.
I am going to try your workaround.
What does it do exactly?

 

[t.lamprecht]

What does it do exactly?

then you can disable the receive and send offload functions as workaround:

As said, it disables some off-loading functions (basically some type of hardware acceleration for sending/receiving packets) of the NIC, that means the host CPU needs to do slightly more work, for the benefit of a working NIC.

 

[lxiosjao]

Thank you very much.

 

[alexdelprete]

To apply it automatically on boot edit /etc/network/interfaces and add a post-up entry for that interface (can also add it on the bridge):

iface eno1 inet manual
    # ... existing lines
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

Thomas, how is the $IFACE variable expanded? There's a post-up script that processes that line and has it defined?

Thanks.

 

[sdet00]

Thomas, how is the $IFACE variable expanded? There's a post-up script that processes that line and has it defined?

Not quire sure what you're asking for here - I just replaced $IFACE with my interface ID, in my case, eno1.

Screenshot that might help: https://i.imgur.com/7POUGnu.jpeg

 

[alexdelprete]

No need to replace it, it's an env variable that contains the interface for the scripts in /etc/network/if-*.d

Every time an interface comes up or goes down, scripts in those dirs are executed and that variable is set with the inteface name.

So now I'm using this, and it works perfectly:

auto lo
iface lo inet loopback


auto eno1
iface eno1 inet static
    address 10.1.10.12/23
    gateway 10.1.10.1
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

 

[sdet00]

Ah, good to know, wasn't aware that this was a universal variable. Very cool. Thank you.

 

[Nicolas Lobariñas]

Hello, I'm having the same issue. Tried the proposed workaround (disabling off-loading) but the issue is still present.
We have a 3 node cluster over Dell R740xd servers. Recently we update from PVE 6.4 to 7.4-16. No changes.
I'm wondering if changing to VirtIO can solve the problem???

# zcat syslog.6.gz | grep -ie kernel -ie keepalived | grep -v Cannot
Aug 18 17:07:20 host kernel: [23593194.904087] e1000 0000:00:15.0 eth1: Reset adapter
Aug 18 17:07:22 host kernel: [23593196.984494] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

# pveversion --verbose
proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-4
pve-kernel-5.4: 6.4-20
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 16.2.13-pve1
ceph-fuse: 16.2.13-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.3-1
proxmox-backup-file-restore: 2.4.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-5
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

# cat /etc/pve/qemu-server/1034.conf 
agent: 1
bootdisk: scsi0
cores: 12
cpu: kvm64,flags=+aes
hotplug: disk,network,usb,cpu
ide2: none,media=cdrom
memory: 2048
name: host
net0: e1000=3E:20:FD:8F:64:5A,bridge=vmbr0,firewall=1,tag=92
net1: e1000=FA:96:D8:E8:9A:0A,bridge=vmbr0,firewall=1,tag=910
net2: e1000=02:63:87:0B:E5:B3,bridge=vmbr0,firewall=1,tag=932
net3: e1000=02:0C:05:5D:7F:7B,bridge=vmbr0,firewall=1,tag=337
numa: 0
onboot: 1
ostype: l26
scsi0: pool:vm-1034-disk-0,discard=on,size=20G
scsihw: virtio-scsi-pci
smbios1: uuid=a9d66c4e-8afd-4af2-b597-50e2450047e8
sockets: 1
vcpus: 6
vmgenid: 6826e577-b999-4c32-aa88-9f93f13d86b5

# dmesg | grep -i ethernet
[    2.411983] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[    2.411989] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.770810] igb 0000:19:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.282897] igb 0000:19:00.1: Intel(R) Gigabit Ethernet Network Connection
[    3.676975] igb 0000:19:00.2: Intel(R) Gigabit Ethernet Network Connection
[    3.746888] igb 0000:19:00.3: Intel(R) Gigabit Ethernet Network Connection

# ethtool -i eno1
driver: igb
version: 5.15.108-1-pve
firmware-version: 1.67, 0x80000faa, 19.5.12
expansion-rom-version: 
bus-info: 0000:19:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -k eno1
Features for eno1:
rx-checksumming: on
tx-checksumming: on
    tx-checksum-ipv4: off [fixed]
    tx-checksum-ip-generic: on
    tx-checksum-ipv6: off [fixed]
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: on
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
    tx-tcp-segmentation: off
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp-mangleid-segmentation: off
    tx-tcp6-segmentation: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

 

[sdet00]

Any idea if this is still a problem in Proxmox 8?