NETDEV WATCHDOG: enp4s0 (igc): transmit queue 0 timed out [Intel I226-V]

[tictag]

Following closely on from Another kernel BUG at lib/dynamic_queue_limits.c:27!, which was recently resolved, this closely related issue remains. This issue has no outwardly noticeable symptoms, but NETDEV WATCHDOG: enp4s0 (igc): transmit queue 0 timed out is regularly being logged in PVE's SysLog with each log instance resulting in a kernel Reset adapter resolution.

Oct 01 17:26:53 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 01 17:26:53 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Oct 01 17:26:53 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Oct 01 17:26:53 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 01 17:26:53 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Oct 01 17:26:53 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Oct 01 17:27:04 pve kernel: ------------[ cut here ]------------
Oct 01 17:27:04 pve kernel: NETDEV WATCHDOG: enp4s0 (igc): transmit queue 0 timed out
Oct 01 17:27:04 pve kernel: WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Oct 01 17:27:04 pve kernel: Modules linked in: ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc binfmt_misc nfnetlink_log nfnetlink snd_hda_codec_hdmi snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda i915 snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core snd_compress x86_pkg_temp_thermal intel_powerclamp coretemp ac97_bus snd_pcm_dmaengine drm_buddy kvm_intel ttm snd_hda_intel drm_display_helper snd_intel_dspcfg snd_intel_sdw_acpi cec kvm snd_hda_codec intel_rapl_msr cmdlinepart rc_core crct10dif_pclmul snd_hda_core polyval_generic ghash_clmulni_intel snd_hwdep processor_thermal_device_pci_legacy sha512_ssse3 drm_kms_helper processor_thermal_device aesni_intel spi_nor snd_pcm processor_thermal_rfim crypto_simd cryptd pcspkr
Oct 01 17:27:04 pve kernel:  intel_cstate processor_thermal_mbox i2c_algo_bit mei_me snd_timer syscopyarea processor_thermal_rapl intel_rapl_common snd sysfillrect int340x_thermal_zone wmi_bmof mtd mei soundcore sysimgblt intel_soc_dts_iosf acpi_tad acpi_pad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb uas usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme xhci_pci spi_intel_pci i2c_i801 xhci_pci_renesas crc32_pclmul spi_intel i2c_smbus sdhci_pci nvme_core xhci_hcd igc cqhci sdhci nvme_common video wmi pinctrl_jasperlake
Oct 01 17:27:04 pve kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P           O       6.2.16-14-pve #1
Oct 01 17:27:04 pve kernel: Hardware name: Default string Default string/Default string, BIOS 5.19 11/11/2022
Oct 01 17:27:04 pve kernel: RIP: 0010:dev_watchdog+0x23a/0x250
[CONTENT SNIPPED]
Oct 01 17:27:04 pve kernel: igc 0000:04:00.0 enp4s0: Reset adapter
Oct 01 17:27:04 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Oct 01 17:27:06 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Oct 01 17:27:06 pve kernel: vmbr0: port 3(enp5s0) entered disabled state


Oct 07 07:38:54 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 07 07:38:54 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Oct 07 07:38:54 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Oct 07 07:38:54 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 07 07:38:54 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Oct 07 07:38:54 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Oct 07 07:39:07 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Oct 07 07:39:07 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Oct 07 07:39:07 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Oct 07 07:39:07 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Oct 07 07:39:07 pve kernel: igc 0000:05:00.0 enp5s0: Register Dump
[CONTENT SNIPPED]
Oct 07 07:39:07 pve kernel: igc 0000:05:00.0 enp5s0: Reset adapter


Oct 07 17:46:54 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 07 17:46:54 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Oct 07 17:46:54 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Oct 07 17:46:55 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 07 17:46:55 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Oct 07 17:46:55 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Oct 07 17:47:05 pve kernel: ------------[ cut here ]------------
Oct 07 17:47:05 pve kernel: NETDEV WATCHDOG: enp5s0 (igc): transmit queue 0 timed out
Oct 07 17:47:05 pve kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Oct 07 17:47:05 pve kernel: Modules linked in: ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc nfnetlink_log binfmt_misc nfnetlink snd_hda_codec_hdmi snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi i915 soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine x86_pkg_temp_thermal snd_hda_intel intel_powerclamp snd_intel_dspcfg coretemp drm_buddy snd_intel_sdw_acpi snd_hda_codec ttm kvm_intel snd_hda_core drm_display_helper intel_rapl_msr kvm cec rc_core snd_hwdep processor_thermal_device_pci_legacy crct10dif_pclmul processor_thermal_device polyval_generic ghash_clmulni_intel drm_kms_helper sha512_ssse3 processor_thermal_rfim snd_pcm aesni_intel processor_thermal_mbox crypto_simd i2c_algo_bit
Oct 07 17:47:05 pve kernel:  cryptd snd_timer processor_thermal_rapl intel_cstate cmdlinepart spi_nor syscopyarea snd intel_rapl_common sysfillrect mei_me int340x_thermal_zone pcspkr wmi_bmof mtd soundcore sysimgblt mei intel_soc_dts_iosf acpi_pad acpi_tad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb uas usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme xhci_pci spi_intel_pci i2c_i801 spi_intel xhci_pci_renesas crc32_pclmul i2c_smbus igc sdhci_pci nvme_core cqhci xhci_hcd nvme_common sdhci video wmi pinctrl_jasperlake
Oct 07 17:47:05 pve kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           O       6.2.16-15-pve #1
Oct 07 17:47:05 pve kernel: Hardware name: Default string Default string/Default string, BIOS 5.19 11/11/2022
Oct 07 17:47:05 pve kernel: RIP: 0010:dev_watchdog+0x23a/0x250
[CONTENT SNIPPED]
Oct 07 17:47:05 pve kernel: igc 0000:05:00.0 enp5s0: Reset adapter
Oct 07 17:47:05 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Oct 07 17:47:08 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Oct 07 17:47:08 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Oct 07 17:47:08 pve kernel: igc 0000:04:00.0 enp4s0: Register Dump
[CONTENT SNIPPED]
Oct 07 17:47:08 pve kernel: igc 0000:04:00.0 enp4s0: Reset adapter


Oct 08 03:57:12 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 08 03:57:12 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Oct 08 03:57:12 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Oct 08 03:57:12 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 08 03:57:12 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Oct 08 03:57:12 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Oct 08 03:57:22 pve kernel: igc 0000:04:00.0 enp4s0: Register Dump
[CONTENT SNIPPED]
Oct 08 03:57:22 pve kernel: igc 0000:04:00.0 enp4s0: Reset adapter
Oct 08 03:57:22 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Oct 08 03:57:25 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Oct 08 03:57:25 pve kernel: vmbr0: port 3(enp5s0) entered disabled state


Oct 13 13:43:13 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Oct 13 13:43:13 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Oct 13 13:43:13 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Oct 13 13:43:14 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Oct 13 14:14:07 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 13 14:14:07 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Oct 13 14:14:07 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Oct 13 14:14:07 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 13 14:14:07 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Oct 13 14:14:07 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Oct 13 14:14:20 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Oct 13 14:14:20 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Oct 13 14:14:20 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Oct 13 14:14:20 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Oct 13 14:14:20 pve kernel: igc 0000:05:00.0 enp5s0: Register Dump
[CONTENT SNIPPED]
Oct 13 14:14:20 pve kernel: igc 0000:05:00.0 enp5s0: Reset adapter


Oct 15 19:29:35 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 15 19:29:35 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Oct 15 19:29:35 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Oct 15 19:29:36 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 15 19:29:36 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Oct 15 19:29:36 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Oct 15 19:29:46 pve kernel: ------------[ cut here ]------------
Oct 15 19:29:46 pve kernel: NETDEV WATCHDOG: enp5s0 (igc): transmit queue 0 timed out
Oct 15 19:29:46 pve kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Oct 15 19:29:46 pve kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc nfnetlink_log binfmt_misc nfnetlink snd_hda_codec_hdmi snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda i915 snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core drm_buddy ttm x86_pkg_temp_thermal intel_powerclamp snd_compress drm_display_helper coretemp ac97_bus cec snd_pcm_dmaengine rc_core kvm_intel snd_hda_intel snd_intel_dspcfg intel_rapl_msr zfs(PO) drm_kms_helper kvm snd_intel_sdw_acpi snd_hda_codec zunicode(PO) crct10dif_pclmul zzstd(O) snd_hda_core zlua(O) snd_hwdep zavl(PO) processor_thermal_device_pci_legacy processor_thermal_device snd_pcm icp(PO) processor_thermal_rfim polyval_generic ghash_clmulni_intel
Oct 15 19:29:46 pve kernel:  processor_thermal_mbox sha512_ssse3 snd_timer processor_thermal_rapl i2c_algo_bit aesni_intel zcommon(PO) cmdlinepart znvpair(PO) syscopyarea intel_rapl_common spi_nor sysfillrect crypto_simd cryptd mei_me snd int340x_thermal_zone spl(O) intel_cstate pcspkr wmi_bmof mtd sysimgblt soundcore mei intel_soc_dts_iosf vhost_net vhost vhost_iotlb tap vfio_pci acpi_tad acpi_pad vfio_pci_core irqbypass vfio_iommu_type1 vfio mac_hid iommufd drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb uas usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci i2c_i801 xhci_pci_renesas nvme spi_intel_pci crc32_pclmul spi_intel i2c_smbus nvme_core video igc xhci_hcd sdhci_pci cqhci nvme_common sdhci wmi pinctrl_jasperlake
Oct 15 19:29:46 pve kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           O       6.2.16-18-pve #1
Oct 15 19:29:46 pve kernel: Hardware name: Default string Default string/Default string, BIOS 5.19 11/11/2022
Oct 15 19:29:46 pve kernel: RIP: 0010:dev_watchdog+0x23a/0x250
[CONTENT SNIPPED]
Oct 15 19:29:46 pve kernel: igc 0000:05:00.0 enp5s0: Reset adapter
Oct 15 19:29:46 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Oct 15 19:29:46 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Oct 15 19:29:47 pve kernel: vmbr0: port 2(enp4s0) entered disabled state


Oct 21 03:40:12 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 21 03:40:12 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Oct 21 03:40:12 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Oct 21 03:40:12 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Oct 21 03:40:12 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Oct 21 03:40:12 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Oct 21 03:40:22 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Oct 21 03:40:22 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Oct 21 03:40:22 pve kernel: igc 0000:04:00.0 enp4s0: Register Dump
[CONTENT SNIPPED]
Oct 21 03:40:22 pve kernel: igc 0000:04:00.0 enp4s0: Reset adapter
Oct 21 03:40:22 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Oct 21 03:40:22 pve kernel: vmbr0: port 2(enp4s0) entered disabled state

Find the complete SysLog (relevant entries) attached.

Output from: nano /etc/network/interfaces

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp3s0 inet manual

iface enp4s0 inet manual

iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.3/24
        gateway 192.168.1.1
        bridge-ports enp3s0 enp4s0 enp5s0
        bridge-stp off
        bridge-fd 0

Output from: lspci -nn | grep Ethernet

02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)

Output from: ethtool -i enp4s0

driver: igc
version: 6.2.16-18-pve
firmware-version: 2017:888d
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Does anybody in the community have any ideas on how to troubleshoot this issue?

David.