e1000 driver hang

Yah, I just run into this on my i3-8109U Nuc, will try the approaches above.

Ony just started doing mass file transfers (transcoding), so I guess that why I didn't run into it before.
 
After two months of dealing with crashes and being unable to solve the problem on some 1L PCs, I tried using a few realtek USB gigabit network adapters. It seems the hosts are still crashing, despite the onboard intel NIC being fully disabled.

Will look more tomorrow. Tried the solutions outlined by others, but haven't found anything that works for me yet.
 
Such a weird error, my home server worked fine for years, and since i switched to a new ISP the problems suddenly starts.. (from cable to Fiber)

if the offloading fix does not work, i just put in a pci-e x1 card i think. Would an Intel 82574L Chipset be good enough and not vunerable for this glitch?
 
Just wanted to confirm that this is a problem with the following card (in a common-as-dirt old higher end Dell desktop), on pve-kernel-5.13:

Code:
# lspci -nnk | grep -A2 Ethernet
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-LM [8086:153a] (rev 04)
        DeviceName:  Onboard LAN
        Subsystem: Dell Ethernet Connection I217-LM [1028:05a4]
        Kernel driver in use: e1000e
        Kernel modules: e1000e

Also, I had to use the 'ethtool' command in /etc/network/interfaces, as the following DID NOT have any effect on the setting as reported by 'ethtool -k'

Code:
iface eno1 inet manual
        offload-tso off
        offload-gso off

Edit: What did work to disable the segmentation offloading for me was to putting this in /etc/network/interfaces as others have done. Note I didn't turn off GRO, only TSO and GSO as my read of the situation tells me turning off only TSO and GSO is the most conservative approach to try first. I haven't tested extensively to see if the problem recurs or not.

Code:
iface eno1 inet manual
        post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"
Same problem here, at first I've tried to disable rx/tx vlan offload but nothing changed, now I'll try with the two/gso and I'll tell you if it works.
I'm on proxmox VE 7.2 on Intel NUC 10th Gen:

Code:
# lspci -v
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (10) I219-V
        DeviceName:  LAN
        Subsystem: Intel Corporation Ethernet Connection (10) I219-V
        Flags: bus master, fast devsel, latency 0, IRQ 145, IOMMU group 11
        Memory at 56300000 (32-bit, non-prefetchable) [size=128K]
        Capabilities: [c8] Power Management version 3
        Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Kernel driver in use: e1000e
        Kernel modules: e1000e

# uname -a
Linux pve 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64 GNU/Linux

UPDATE: it works for me, issue disappeared disabling only gso and tso
 
Last edited:
to confirm, to fix this, you added the below code to /etc/network/interfaces?

Code:
iface eno1 inet manual
        offload-tso off
        offload-gso off
No, I didn’t try that way and I directly used the post-up command

Code:
iface eno1 inet manual
        post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"
 
Last edited:
No, I didn’t try that way and I directly used the post-up command

Code:
iface eno1 inet manual
        post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"

Thanks. I've done that and rebooted the host. It did not fix it for me. I've attached logging and the interfaces(.txt) file.
How can I verify TSO and GSO are succesfully disabled?
 

Attachments

  • eno1.txt
    8.1 KB · Views: 21
  • interfaces.txt
    1.8 KB · Views: 16
Thanks. I've done that and rebooted the host. It did not fix it for me. I've attached logging and the interfaces(.txt) file.
How can I verify TSO and GSO are succesfully disabled?
Someone said that TSO and GSO were not sufficient and disabled also GRO.

Here you can see if segmentation parameters are off:

Bash:
# ethtool -k eno1 | grep -i segmentation                                                                                                                                                         
tcp-segmentation-offload: off                                                                                                                                                                              
        tx-tcp-segmentation: off                                                                                                                                                                           
        tx-tcp-ecn-segmentation: off [fixed]                                                                                                                                                               
        tx-tcp-mangleid-segmentation: off                                                                                                                                                                  
        tx-tcp6-segmentation: off                                                                                                                                                                          
generic-segmentation-offload: off                                                                                                                                                                          
tx-fcoe-segmentation: off [fixed]                                                                                                                                                                          
tx-gre-segmentation: off [fixed]                                                                                                                                                                           
tx-gre-csum-segmentation: off [fixed]                                                                                                                                                                      
tx-ipxip4-segmentation: off [fixed]                                                                                                                                                                        
tx-ipxip6-segmentation: off [fixed]                                                                                                                                                                        
tx-udp_tnl-segmentation: off [fixed]                                                                                                                                                                       
tx-udp_tnl-csum-segmentation: off [fixed]                                                                                                                                                                  
tx-tunnel-remcsum-segmentation: off [fixed]                                                                                                                                                                
tx-sctp-segmentation: off [fixed]                                                                                                                                                                          
tx-esp-segmentation: off [fixed]                                                                                                                                                                           
tx-udp-segmentation: off [fixed]

For example you can try disabling vlan rx/tx offload or GRO too and see if it helps. On my setup the following are still on but give no issues:

Bash:
# ethtool -k eno1 | grep -i ": on"                                                                                                                                                              
rx-checksumming: on                                                                                                                                                                                        
tx-checksumming: on                                                                                                                                                                                        
        tx-checksum-ip-generic: on                                                                                                                                                                        
scatter-gather: on                                                                                                                                                                                        
        tx-scatter-gather: on                                                                                                                                                                              
generic-receive-offload: on                                                                                                                                                                                
rx-vlan-offload: on                                                                                                                                                                                        
tx-vlan-offload: on                                                                                                                                                                                        
receive-hashing: on                                                                                                                                                                                        
highdma: on [fixed]

GRO is the generic-receive-offload above.

You can try also disabling parameters interactively, in order to see if you're allowed to disable them and then use the command ethtool -k eno1 to see if they've been correctly disabled. If it works and the problem disappear, update the command in the /etc/network/interfaces post-up section.

To disable rx-vlan-offload for example you can run ethtool -K eno1 rxvlan off. The parameters are named with shortnames that you find in the man page:

Code:
       -K --features --offload
              Changes the offload parameters and other features of the specified network device.  The following feature names are built-in and others may be defined by the kernel.

           rx on|off
                  Specifies whether RX checksumming should be enabled.

           tx on|off
                  Specifies whether TX checksumming should be enabled.

           sg on|off
                  Specifies whether scatter-gather should be enabled.

           tso on|off
                  Specifies whether TCP segmentation offload should be enabled.

           ufo on|off
                  Specifies whether UDP fragmentation offload should be enabled

           gso on|off
                  Specifies whether generic segmentation offload should be enabled

           gro on|off
                  Specifies whether generic receive offload should be enabled

           lro on|off
                  Specifies whether large receive offload should be enabled

           rxvlan on|off
                  Specifies whether RX VLAN acceleration should be enabled

           txvlan on|off
                  Specifies whether TX VLAN acceleration should be enabled

           ntuple on|off
                  Specifies whether Rx ntuple filters and actions should be enabled

           rxhash on|off
                  Specifies whether receive hashing offload should be enabled
 
Last edited:
Someone said that TSO and GSO were not sufficient and disabled also GRO.

Here you can see if segmentation parameters are off:

Bash:
# ethtool -k eno1 | grep -i segmentation                                                                                                                                                        
tcp-segmentation-offload: off                                                                                                                                                                             
        tx-tcp-segmentation: off                                                                                                                                                                          
        tx-tcp-ecn-segmentation: off [fixed]                                                                                                                                                              
        tx-tcp-mangleid-segmentation: off                                                                                                                                                                 
        tx-tcp6-segmentation: off                                                                                                                                                                         
generic-segmentation-offload: off                                                                                                                                                                         
tx-fcoe-segmentation: off [fixed]                                                                                                                                                                         
tx-gre-segmentation: off [fixed]                                                                                                                                                                          
tx-gre-csum-segmentation: off [fixed]                                                                                                                                                                     
tx-ipxip4-segmentation: off [fixed]                                                                                                                                                                       
tx-ipxip6-segmentation: off [fixed]                                                                                                                                                                       
tx-udp_tnl-segmentation: off [fixed]                                                                                                                                                                      
tx-udp_tnl-csum-segmentation: off [fixed]                                                                                                                                                                 
tx-tunnel-remcsum-segmentation: off [fixed]                                                                                                                                                               
tx-sctp-segmentation: off [fixed]                                                                                                                                                                         
tx-esp-segmentation: off [fixed]                                                                                                                                                                          
tx-udp-segmentation: off [fixed]

For example you can try disabling vlan rx/tx offload or GRO too and see if it helps. On my setup the following are still on but give no issues:

Bash:
# ethtool -k eno1 | grep -i ": on"                                                                                                                                                             
rx-checksumming: on                                                                                                                                                                                       
tx-checksumming: on                                                                                                                                                                                       
        tx-checksum-ip-generic: on                                                                                                                                                                       
scatter-gather: on                                                                                                                                                                                       
        tx-scatter-gather: on                                                                                                                                                                             
generic-receive-offload: on                                                                                                                                                                               
rx-vlan-offload: on                                                                                                                                                                                       
tx-vlan-offload: on                                                                                                                                                                                       
receive-hashing: on                                                                                                                                                                                       
highdma: on [fixed]

GRO is the generic-receive-offload above.

You can try also disabling parameters interactively, in order to see if you're allowed to disable them and then use the command ethtool -k eno1 to see if they've been correctly disabled. If it works and the problem disappear, update the command in the /etc/network/interfaces post-up section.

To disable rx-vlan-offload for example you can run ethtool -K eno1 rxvlan off. The parameters are named with shortnames that you find in the man page:

Code:
       -K --features --offload
              Changes the offload parameters and other features of the specified network device.  The following feature names are built-in and others may be defined by the kernel.

           rx on|off
                  Specifies whether RX checksumming should be enabled.

           tx on|off
                  Specifies whether TX checksumming should be enabled.

           sg on|off
                  Specifies whether scatter-gather should be enabled.

           tso on|off
                  Specifies whether TCP segmentation offload should be enabled.

           ufo on|off
                  Specifies whether UDP fragmentation offload should be enabled

           gso on|off
                  Specifies whether generic segmentation offload should be enabled

           gro on|off
                  Specifies whether generic receive offload should be enabled

           lro on|off
                  Specifies whether large receive offload should be enabled

           rxvlan on|off
                  Specifies whether RX VLAN acceleration should be enabled

           txvlan on|off
                  Specifies whether TX VLAN acceleration should be enabled

           ntuple on|off
                  Specifies whether Rx ntuple filters and actions should be enabled

           rxhash on|off
                  Specifies whether receive hashing offload should be enabled

Thanks for the instructions. I'll make changes and report back
 
  • Like
Reactions: MightySlaytanic
Quite frustrating that this problem is still ongoing today... I had this issue around a year or two back, and put together a new Proxmox system (well, it's refurbished) and forgot about this bug. Workarounds did the trick thankfully.
 
  • Like
Reactions: MightySlaytanic
stable but with a significant drop in performance with

iface eno1 inet manual
pre-up /usr/sbin/ethtool -K eno1 tso off gso off

Anyone able to get the "e1000e 0000:00:1f.6 eno1: Detected Hardware Unit" worked around w/o performance degradation ?

 
stable but with a significant drop in performance with

iface eno1 inet manual
pre-up /usr/sbin/ethtool -K eno1 tso off gso off

Anyone able to get the "e1000e 0000:00:1f.6 eno1: Detected Hardware Unit" worked around w/o performance degradation ?
What kind of degradation do you experience? I have those two parameters disabled and I’ve tested about 930Mbps on the 1Gbps e1000 interface

Code:
grep post /etc/network/interfaces
        post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"

Code:
iperf3 -c 10.0.0.3 -i 10 -p 5201 -t 10s -w 416k -P 4
Connecting to host 10.0.0.3, port 5201
[  5] local 10.0.0.4 port 43410 connected to 10.0.0.3 port 5201
[  7] local 10.0.0.4 port 43424 connected to 10.0.0.3 port 5201
[  9] local 10.0.0.4 port 43434 connected to 10.0.0.3 port 5201
[ 11] local 10.0.0.4 port 43444 connected to 10.0.0.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec   285 MBytes   239 Mbits/sec    0    458 KBytes     
[  7]   0.00-10.00  sec   276 MBytes   231 Mbits/sec    0    451 KBytes     
[  9]   0.00-10.00  sec   276 MBytes   232 Mbits/sec    0    452 KBytes     
[ 11]   0.00-10.00  sec   276 MBytes   231 Mbits/sec    0    450 KBytes     
[SUM]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec    0           
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   285 MBytes   239 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   285 MBytes   239 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   276 MBytes   231 Mbits/sec    0             sender
[  7]   0.00-10.00  sec   275 MBytes   231 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   276 MBytes   232 Mbits/sec    0             sender
[  9]   0.00-10.00  sec   276 MBytes   231 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   276 MBytes   231 Mbits/sec    0             sender
[ 11]   0.00-10.00  sec   276 MBytes   231 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec    0             sender
[SUM]   0.00-10.00  sec  1.09 GBytes   933 Mbits/sec                  receiver
 
Last edited:
  • Like
Reactions: guiambros
What kind of degradation do you experience? I have those two parameters disabled and I’ve tested about 930Mbps on the 1Gbps e1000 interface

Code:
grep post /etc/network/interfaces
        post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"

Code:
iperf3 -c 10.0.0.3 -i 10 -p 5201 -t 10s -w 416k -P 4
Connecting to host 10.0.0.3, port 5201
[  5] local 10.0.0.4 port 43410 connected to 10.0.0.3 port 5201
[  7] local 10.0.0.4 port 43424 connected to 10.0.0.3 port 5201
[  9] local 10.0.0.4 port 43434 connected to 10.0.0.3 port 5201
[ 11] local 10.0.0.4 port 43444 connected to 10.0.0.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec   285 MBytes   239 Mbits/sec    0    458 KBytes    
[  7]   0.00-10.00  sec   276 MBytes   231 Mbits/sec    0    451 KBytes    
[  9]   0.00-10.00  sec   276 MBytes   232 Mbits/sec    0    452 KBytes    
[ 11]   0.00-10.00  sec   276 MBytes   231 Mbits/sec    0    450 KBytes    
[SUM]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec    0          
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   285 MBytes   239 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   285 MBytes   239 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   276 MBytes   231 Mbits/sec    0             sender
[  7]   0.00-10.00  sec   275 MBytes   231 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   276 MBytes   232 Mbits/sec    0             sender
[  9]   0.00-10.00  sec   276 MBytes   231 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   276 MBytes   231 Mbits/sec    0             sender
[ 11]   0.00-10.00  sec   276 MBytes   231 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec    0             sender
[SUM]   0.00-10.00  sec  1.09 GBytes   933 Mbits/sec                  receiver
I'm also getting 930Mbps - moving on to debug the router/ISP settings.
I appreciate the help
Thank you
 
I always have to reboot my machine twice because of this. None of the fixes seems to work 100% for me. Like today, my PVE was online for 38 days. I've ran my updates and everytime again, 5 minutes after a reboot, my pve freezes. Hard power off + power on reboots it and it is able to stay online for weeks/months/... Only the first warm reboot after updates freezes after 5 minutes every time again.

So my procedure is:
> Running for weeks/months
> Update PVE
> Reboot node
> FREEZES after 5 minutes
> Hard power off + power on (using a Sonoff now because I'm lazy)
> Boots up autom. because of boot on power setting
> Running for weeks/months
 
Last edited:
I was having this issue with the interface being reset all the time under heavy load.

Here is the error:

Code:
[Fri May 14 23:55:54 2021] ------------[ cut here ]------------
[Fri May 14 23:55:54 2021] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
[Fri May 14 23:55:54 2021] WARNING: CPU: 12 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x264/0x270
[Fri May 14 23:55:54 2021] Modules linked in: veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw softdog ip6table_mangle ip6table_filter ip6_tables xt_conntrack xt_tcpudp xt_nat xt_MASQUERADE iptable_nat nf_nat nfnetlink_log bpfilter nfnetlink intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass rapl intel_cstate input_leds serio_raw wmi_bmof intel_wmi_thunderbolt intel_pch_thermal acpi_pad mac_hid vhost_net vhost tap coretemp sunrpc autofs4 btrfs zstd_compress dm_crypt raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid0 multipath linear xt_comment xt_recent xt_connlimit nf_conncount xt_state nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_length xt_hl xt_tcpmss xt_TCPMSS ipt_REJECT nf_reject_ipv4 xt_dscp xt_multiport xt_limit iptable_mangle iptable_filter ip_tables x_tables bfq raid1 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper ahci xhci_pci e1000e i2c_i801
[Fri May 14 23:55:54 2021]  libahci xhci_hcd wmi video pinctrl_cannonlake pinctrl_intel
[Fri May 14 23:55:54 2021] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 5.4.114-1-pve #1
[Fri May 14 23:55:54 2021] Hardware name: Gigabyte Technology Co., Ltd. B360 HD3P-LM/B360HD3PLM-CF, BIOS F4 HZ 04/30/2019
[Fri May 14 23:55:54 2021] RIP: 0010:dev_watchdog+0x264/0x270
[Fri May 14 23:55:54 2021] Code: 48 85 c0 75 e6 eb a0 4c 89 ef c6 05 80 c8 ef 00 01 e8 20 b8 fa ff 89 d9 4c 89 ee 48 c7 c7 98 5c c3 92 48 89 c2 e8 c5 56 15 00 <0f> 0b eb 82 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41
[Fri May 14 23:55:54 2021] RSP: 0018:ffff9decc03d8e58 EFLAGS: 00010282
[Fri May 14 23:55:54 2021] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000083f
[Fri May 14 23:55:54 2021] RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
[Fri May 14 23:55:54 2021] RBP: ffff9decc03d8e88 R08: 00000000000003a4 R09: ffffffff9339e768
[Fri May 14 23:55:54 2021] R10: 0000000000000774 R11: ffff9decc03d8cb0 R12: 0000000000000001
[Fri May 14 23:55:54 2021] R13: ffff925deb2a8000 R14: ffff925deb2a8480 R15: ffff925deb1ee880
[Fri May 14 23:55:54 2021] FS:  0000000000000000(0000) GS:ffff925dff300000(0000) knlGS:0000000000000000
[Fri May 14 23:55:54 2021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri May 14 23:55:54 2021] CR2: 00007f38443ebbc8 CR3: 0000000e649e6003 CR4: 00000000003606e0
[Fri May 14 23:55:54 2021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[Fri May 14 23:55:54 2021] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[Fri May 14 23:55:54 2021] Call Trace:
[Fri May 14 23:55:54 2021]  <IRQ>
[Fri May 14 23:55:54 2021]  ? pfifo_fast_enqueue+0x160/0x160
[Fri May 14 23:55:54 2021]  call_timer_fn+0x32/0x130
[Fri May 14 23:55:54 2021]  run_timer_softirq+0x1a5/0x430
[Fri May 14 23:55:54 2021]  ? ktime_get+0x3c/0xa0
[Fri May 14 23:55:54 2021]  ? lapic_next_deadline+0x2c/0x40
[Fri May 14 23:55:54 2021]  ? clockevents_program_event+0x93/0xf0
[Fri May 14 23:55:54 2021]  __do_softirq+0xdc/0x2d4
[Fri May 14 23:55:54 2021]  irq_exit+0xa9/0xb0
[Fri May 14 23:55:54 2021]  smp_apic_timer_interrupt+0x79/0x130
[Fri May 14 23:55:54 2021]  apic_timer_interrupt+0xf/0x20
[Fri May 14 23:55:54 2021]  </IRQ>
[Fri May 14 23:55:54 2021] RIP: 0010:cpuidle_enter_state+0xbd/0x450
[Fri May 14 23:55:54 2021] Code: ff e8 b7 79 88 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 63 03 00 00 31 ff e8 ba 81 8e ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 8d 02 00 00 49 63 cd 48 8b 75 d0 48 2b 75 c8 48 8d
[Fri May 14 23:55:54 2021] RSP: 0018:ffff9decc0147e48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[Fri May 14 23:55:54 2021] RAX: ffff925dff32ae00 RBX: ffffffff92f57c40 RCX: 000000000000001f
[Fri May 14 23:55:54 2021] RDX: 000002c9a813f813 RSI: 00000000238e3d6b RDI: 0000000000000000
[Fri May 14 23:55:54 2021] RBP: ffff9decc0147e88 R08: 0000000000000002 R09: 000000000002a680
[Fri May 14 23:55:54 2021] R10: 00000a21d04c5df8 R11: ffff925dff329aa0 R12: ffffbdecbfd16f08
[Fri May 14 23:55:54 2021] R13: 0000000000000001 R14: ffffffff92f57cb8 R15: ffffffff92f57ca0
[Fri May 14 23:55:54 2021]  ? cpuidle_enter_state+0x99/0x450
[Fri May 14 23:55:54 2021]  cpuidle_enter+0x2e/0x40
[Fri May 14 23:55:54 2021]  call_cpuidle+0x23/0x40
[Fri May 14 23:55:54 2021]  do_idle+0x22c/0x270
[Fri May 14 23:55:54 2021]  cpu_startup_entry+0x1d/0x20
[Fri May 14 23:55:54 2021]  start_secondary+0x166/0x1c0
[Fri May 14 23:55:54 2021]  secondary_startup_64+0xa4/0xb0
[Fri May 14 23:55:54 2021] ---[ end trace ab9792688d4e93f4 ]---
[Fri May 14 23:55:54 2021] e1000e 0000:00:1f.6 eth0: Reset adapter unexpectedly
[Fri May 14 23:56:00 2021] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Fri May 14 23:58:08 2021] e1000e 0000:00:1f.6 eth0: Reset adapter unexpectedly
[Fri May 14 23:58:13 2021] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Sat May 15 00:08:17 2021] e1000e 0000:00:1f.6 eth0: Reset adapter unexpectedly
[Sat May 15 00:08:22 2021] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Sat May 15 00:08:33 2021] e1000e 0000:00:1f.6 eth0: Reset adapter unexpectedly

It happens on kernels:

* Linux version 5.4.114-1-pve (build@proxmox) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200) ()
* Linux version 5.11.7-1-pve (build@pve) (gcc (Debian 8.3.0-6) 8.3.0, GNU ld (GNU Binutils for Debian) 2.31.1) #1 SMP PVE 5.11.7-1~bpo10 (Thu, 18 Mar 2021 16:17:24 +0100) ()

I have this NIC:
Code:
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)

but it might happen as well on any other one related.

I've tried settings various kernel options in /etc/default/grub, e.g.:
Code:
pcie_aspm=off
but it didn't help.

The only workaround here is (replace eth0 with your interface name):

Code:
apt install -y ethtool
ethtool -K eth0 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off

to make this permanent just add this into your /etc/network/interfaces:
Code:
auto eth0
iface eth0 inet static
  offload-gso off
  offload-gro off
  offload-tso off
  offload-rx off
  offload-tx off
  offload-rxvlan off
  offload-txvlan off
  offload-sg off
  offload-ufo off
  offload-lro off
  address x.x.x.x
  netmask a.a.a.a
  gateway z.z.z.z

NOTE: only disabling tso or gso doesn't help in my case I had to disable all offloading!
my current /etc/network/interfaces file looks like this
Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
  address 192.168.1.61/24
  gateway 192.168.1.1
  bridge-ports eno1
  bridge-stp off
  bridge-fd 0
should I add the `offline` stuff under eno1 or vmbr0? I'm a bit confused since the bridge one has the ip. Thanks.
 
my current /etc/network/interfaces file looks like this
Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
  address 192.168.1.61/24
  gateway 192.168.1.1
  bridge-ports eno1
  bridge-stp off
  bridge-fd 0
should I add the `offline` stuff under eno1 or vmbr0? I'm a bit confused since the bridge one has the ip. Thanks.
Add it under the physical interface eno1. I have this under eno1:
Code:
iface eno1 inet manual
    post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"
 
Last edited:
The only workaround here is (replace eth0 with your interface name):

Code:
apt install -y ethtool
ethtool -K eth0 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off

to make this permanent just add this into your /etc/network/interfaces:
Code:
auto eth0
iface eth0 inet static
  offload-gso off
  offload-gro off
  offload-tso off
  offload-rx off
  offload-tx off
  offload-rxvlan off
  offload-txvlan off
  offload-sg off
  offload-ufo off
  offload-lro off
  address x.x.x.x
  netmask a.a.a.a
  gateway z.z.z.z

NOTE: only disabling tso or gso doesn't help in my case I had to disable all offloading!

Thank you! Was driving me crazy the network drop outs
 
Hi, the ethtool fix works fine, and I've set-up a post-up action to run in each time eno1 is up.
But after adding a new network interface on one of my vm with the VLAN tag 20, the host freeze issue returns.

This is my current interfaces file:
Bash:
auto lo
iface lo inet loopback

iface eno1 inet manual
        post-up /sbin/ethtool -K $IFACE tso off gso off gro off

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.3/24
        gateway 192.168.10.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

But apparently, ProxmoxVE creates a new linux interface dynamically to add/manage the VLAN ... and ethtool fixes must also be applied on it.
Bash:
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 2c:f0:5d:db:ce:81  txqueuelen 1000  (Ethernet)
        RX packets 45641515  bytes 56862382902 (52.9 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 23033002  bytes 14289197449 (13.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xa4500000-a4520000

eno1.20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 2c:f0:5d:db:ce:81  txqueuelen 1000  (Ethernet)
        RX packets 1788798  bytes 218498559 (208.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3137555  bytes 4349237578 (4.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

... tapxxx/fwxx interfaces removed

vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.10.3  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::2ef0:5dff:fedb:ce81  prefixlen 64  scopeid 0x20<link>
        ether 2c:f0:5d:db:ce:81  txqueuelen 1000  (Ethernet)
        RX packets 38069900  bytes 55406252647 (51.6 GiB)
        RX errors 0  dropped 215192  overruns 0  frame 0
        TX packets 4793735  bytes 464390058 (442.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr0v20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 9e:e3:f7:28:36:2f  txqueuelen 1000  (Ethernet)
        RX packets 77057  bytes 6628091 (6.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Currently I'm running ethtool manually on the eno1.20 VLAN interface after each restart (and that is fixing the host freeze issue again), but I would like it to be done automatically like for eno1.

So what is the proper way to add ethtool post-up to interfaces for eno1.20 ?
Is there will be conflicts with the fact that Proxmox already manage it internally ?
 
  • Like
Reactions: sdettmer

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!