Hey guys, I'm getting this log message:
Feb 10 07:01:54 pvehouston1 kernel: ------------[ cut here ]------------Feb 10 07:01:54 pvehouston1 kernel: NETDEV WATCHDOG: enp6s0 (igb): transmit queue 3 timed out 9080 msFeb 10 07:01:54 pvehouston1 kernel: WARNING: CPU: 7 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x260/0x270Feb 10 07:01:54 pvehouston1 kernel: Modules linked in: tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables sunrpc bonding tls softdog nfnetlink_log nfnetlink binfmt_misc intel_rapl_msr intel_rapl_common amdgpu snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof edac_mce_amd snd_sof_utils snd_hda_codec_hdmi snd_soc_core kvm_amd snd_hda_intel amdxcp ipmi_ssif snd_intel_dspcfg iommu_v2 snd_compress drm_buddy snd_intel_sdw_acpi kvm gpu_sched ac97_bus snd_hda_codec drm_suballoc_helper snd_pcm_dmaengine drm_ttm_helper snd_pci_ps ttm irqbypass snd_rpl_pci_acp6x snd_hda_core crct10dif_pclmul snd_acp_pci polyval_clmulni polyval_generic snd_pci_acp6x snd_hwdep ghash_clmulni_intel drm_display_helper aesni_intel snd_pcm cec ast snd_pci_acp5x snd_timer crypto_simd rc_core drm_shmem_helper snd_rn_pci_acp3x cryptd snd snd_acp_config snd_soc_acpi acpi_ipmi rapl soundcore drm_kms_helper pcspkr wmi_bmof snd_pci_acp3x ccp k10tempFeb 10 07:01:54 pvehouston1 kernel: ipmi_si ipmi_devintf ipmi_msghandler joydev input_leds mac_hid vhost_net vhost vhost_iotlb tap efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 usbkbd hid_generic usbmouse usbhid hid cdc_ether usbnet mii uas usb_storage zfs(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb xhci_pci xhci_pci_renesas nvme crc32_pclmul igb xhci_hcd bnxt_en ahci nvme_core i2c_piix4 i2c_algo_bit libahci dca nvme_common video wmi gpio_amdptFeb 10 07:01:54 pvehouston1 kernel: CPU: 7 PID: 0 Comm: swapper/7 Tainted: P W O 6.5.11-8-pve #1Feb 10 07:01:54 pvehouston1 kernel: Hardware name: ASRockRack 1U4LW-B650/2L2T/B650D4U-2L2T/BCM, BIOS 3.17 05/24/2023Feb 10 07:01:54 pvehouston1 kernel: RIP: 0010:dev_watchdog+0x260/0x270Feb 10 07:01:54 pvehouston1 kernel: Code: ff ff 48 89 df c6 05 37 3b 78 01 01 e8 b9 80 f9 ff 44 8b 45 cc 44 89 f9 48 89 de 48 89 c2 48 c7 c7 b0 9e 83 a8 e8 30 ce 33 ff <0f> 0b e9 1d ff ff ff 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90Feb 10 07:01:54 pvehouston1 kernel: RSP: 0018:ffffb63740424e40 EFLAGS: 00010246Feb 10 07:01:54 pvehouston1 kernel: RAX: 0000000000000000 RBX: ffffa01395ed4000 RCX: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: RBP: ffffb63740424e78 R08: 0000000000000000 R09: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa01395ed44c8Feb 10 07:01:54 pvehouston1 kernel: R13: ffffa01395ed441c R14: 0000000000000000 R15: 0000000000000003Feb 10 07:01:54 pvehouston1 kernel: FS: 0000000000000000(0000) GS:ffffa032281c0000(0000) knlGS:0000000000000000Feb 10 07:01:54 pvehouston1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033Feb 10 07:01:54 pvehouston1 kernel: CR2: 000055809cd3a388 CR3: 00000008e5634000 CR4: 0000000000750ee0Feb 10 07:01:54 pvehouston1 kernel: PKRU: 55555554Feb 10 07:01:54 pvehouston1 kernel: Call Trace:Feb 10 07:01:54 pvehouston1 kernel: <IRQ>Feb 10 07:01:54 pvehouston1 kernel: ? show_regs+0x6d/0x80Feb 10 07:01:54 pvehouston1 kernel: ? __warn+0x89/0x160Feb 10 07:01:54 pvehouston1 kernel: ? dev_watchdog+0x260/0x270Feb 10 07:01:54 pvehouston1 kernel: ? report_bug+0x17e/0x1b0Feb 10 07:01:54 pvehouston1 kernel: ? irq_work_queue+0x2f/0x70Feb 10 07:01:54 pvehouston1 kernel: ? handle_bug+0x46/0x90Feb 10 07:01:54 pvehouston1 kernel: ? exc_invalid_op+0x18/0x80Feb 10 07:01:54 pvehouston1 kernel: ? asm_exc_invalid_op+0x1b/0x20Feb 10 07:01:54 pvehouston1 kernel: ? dev_watchdog+0x260/0x270Feb 10 07:01:54 pvehouston1 kernel: ? __pfx_dev_watchdog+0x10/0x10Feb 10 07:01:54 pvehouston1 kernel: call_timer_fn+0x29/0x160Feb 10 07:01:54 pvehouston1 kernel: ? __pfx_dev_watchdog+0x10/0x10Feb 10 07:01:54 pvehouston1 kernel: __run_timers+0x259/0x310Feb 10 07:01:54 pvehouston1 kernel: run_timer_softirq+0x1d/0x40Feb 10 07:01:54 pvehouston1 kernel: __do_softirq+0xd1/0x303Feb 10 07:01:54 pvehouston1 kernel: __irq_exit_rcu+0x75/0xa0Feb 10 07:01:54 pvehouston1 kernel: irq_exit_rcu+0xe/0x20Feb 10 07:01:54 pvehouston1 kernel: sysvec_apic_timer_interrupt+0x92/0xd0Feb 10 07:01:54 pvehouston1 kernel: </IRQ>Feb 10 07:01:54 pvehouston1 kernel: <TASK>Feb 10 07:01:54 pvehouston1 kernel: asm_sysvec_apic_timer_interrupt+0x1b/0x20Feb 10 07:01:54 pvehouston1 kernel: RIP: 0010
oll_idle+0x5d/0xb5Feb 10 07:01:54 pvehouston1 kernel: Code: 4f 02 20 49 8b 07 a8 08 75 32 4c 89 ef 48 89 de e8 38 ff ff ff 49 89 c5 b8 c9 00 00 00 49 8b 17 83 e2 08 75 17 f3 90 83 e8 01 <75> f1 e8 dc de ff ff 4c 29 e0 49 39 c5 73 df 80 0b 04 fa 0f 1f 44Feb 10 07:01:54 pvehouston1 kernel: RSP: 0018:ffffb637401efe18 EFLAGS: 00000206Feb 10 07:01:54 pvehouston1 kernel: RAX: 0000000000000082 RBX: ffffa01388b59400 RCX: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: RBP: ffffb637401efe40 R08: 0000000000000000 R09: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00008a6bfb508992Feb 10 07:01:54 pvehouston1 kernel: R13: 0000000000008ca0 R14: 0000000000000000 R15: ffffa013808bc000Feb 10 07:01:54 pvehouston1 kernel: ? poll_idle+0x64/0xb5Feb 10 07:01:54 pvehouston1 kernel: cpuidle_enter_state+0x82/0x470Feb 10 07:01:54 pvehouston1 kernel: cpuidle_enter+0x2e/0x50Feb 10 07:01:54 pvehouston1 kernel: call_cpuidle+0x23/0x60Feb 10 07:01:54 pvehouston1 kernel: do_idle+0x202/0x260Feb 10 07:01:54 pvehouston1 kernel: cpu_startup_entry+0x2a/0x30Feb 10 07:01:54 pvehouston1 kernel: start_secondary+0x119/0x140Feb 10 07:01:54 pvehouston1 kernel: secondary_startup_64_no_verify+0x17e/0x18bFeb 10 07:01:54 pvehouston1 kernel: </TASK>Feb 10 07:01:54 pvehouston1 kernel: ---[ end trace 0000000000000000 ]---
I tried updating my network driver, checking my network hardware and investigating my network traffic to see if there's a congestion issue. What should I be doing? Where should I be looking at?
Feb 10 07:01:54 pvehouston1 kernel: ------------[ cut here ]------------Feb 10 07:01:54 pvehouston1 kernel: NETDEV WATCHDOG: enp6s0 (igb): transmit queue 3 timed out 9080 msFeb 10 07:01:54 pvehouston1 kernel: WARNING: CPU: 7 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x260/0x270Feb 10 07:01:54 pvehouston1 kernel: Modules linked in: tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables sunrpc bonding tls softdog nfnetlink_log nfnetlink binfmt_misc intel_rapl_msr intel_rapl_common amdgpu snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof edac_mce_amd snd_sof_utils snd_hda_codec_hdmi snd_soc_core kvm_amd snd_hda_intel amdxcp ipmi_ssif snd_intel_dspcfg iommu_v2 snd_compress drm_buddy snd_intel_sdw_acpi kvm gpu_sched ac97_bus snd_hda_codec drm_suballoc_helper snd_pcm_dmaengine drm_ttm_helper snd_pci_ps ttm irqbypass snd_rpl_pci_acp6x snd_hda_core crct10dif_pclmul snd_acp_pci polyval_clmulni polyval_generic snd_pci_acp6x snd_hwdep ghash_clmulni_intel drm_display_helper aesni_intel snd_pcm cec ast snd_pci_acp5x snd_timer crypto_simd rc_core drm_shmem_helper snd_rn_pci_acp3x cryptd snd snd_acp_config snd_soc_acpi acpi_ipmi rapl soundcore drm_kms_helper pcspkr wmi_bmof snd_pci_acp3x ccp k10tempFeb 10 07:01:54 pvehouston1 kernel: ipmi_si ipmi_devintf ipmi_msghandler joydev input_leds mac_hid vhost_net vhost vhost_iotlb tap efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 usbkbd hid_generic usbmouse usbhid hid cdc_ether usbnet mii uas usb_storage zfs(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb xhci_pci xhci_pci_renesas nvme crc32_pclmul igb xhci_hcd bnxt_en ahci nvme_core i2c_piix4 i2c_algo_bit libahci dca nvme_common video wmi gpio_amdptFeb 10 07:01:54 pvehouston1 kernel: CPU: 7 PID: 0 Comm: swapper/7 Tainted: P W O 6.5.11-8-pve #1Feb 10 07:01:54 pvehouston1 kernel: Hardware name: ASRockRack 1U4LW-B650/2L2T/B650D4U-2L2T/BCM, BIOS 3.17 05/24/2023Feb 10 07:01:54 pvehouston1 kernel: RIP: 0010:dev_watchdog+0x260/0x270Feb 10 07:01:54 pvehouston1 kernel: Code: ff ff 48 89 df c6 05 37 3b 78 01 01 e8 b9 80 f9 ff 44 8b 45 cc 44 89 f9 48 89 de 48 89 c2 48 c7 c7 b0 9e 83 a8 e8 30 ce 33 ff <0f> 0b e9 1d ff ff ff 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90Feb 10 07:01:54 pvehouston1 kernel: RSP: 0018:ffffb63740424e40 EFLAGS: 00010246Feb 10 07:01:54 pvehouston1 kernel: RAX: 0000000000000000 RBX: ffffa01395ed4000 RCX: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: RBP: ffffb63740424e78 R08: 0000000000000000 R09: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa01395ed44c8Feb 10 07:01:54 pvehouston1 kernel: R13: ffffa01395ed441c R14: 0000000000000000 R15: 0000000000000003Feb 10 07:01:54 pvehouston1 kernel: FS: 0000000000000000(0000) GS:ffffa032281c0000(0000) knlGS:0000000000000000Feb 10 07:01:54 pvehouston1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033Feb 10 07:01:54 pvehouston1 kernel: CR2: 000055809cd3a388 CR3: 00000008e5634000 CR4: 0000000000750ee0Feb 10 07:01:54 pvehouston1 kernel: PKRU: 55555554Feb 10 07:01:54 pvehouston1 kernel: Call Trace:Feb 10 07:01:54 pvehouston1 kernel: <IRQ>Feb 10 07:01:54 pvehouston1 kernel: ? show_regs+0x6d/0x80Feb 10 07:01:54 pvehouston1 kernel: ? __warn+0x89/0x160Feb 10 07:01:54 pvehouston1 kernel: ? dev_watchdog+0x260/0x270Feb 10 07:01:54 pvehouston1 kernel: ? report_bug+0x17e/0x1b0Feb 10 07:01:54 pvehouston1 kernel: ? irq_work_queue+0x2f/0x70Feb 10 07:01:54 pvehouston1 kernel: ? handle_bug+0x46/0x90Feb 10 07:01:54 pvehouston1 kernel: ? exc_invalid_op+0x18/0x80Feb 10 07:01:54 pvehouston1 kernel: ? asm_exc_invalid_op+0x1b/0x20Feb 10 07:01:54 pvehouston1 kernel: ? dev_watchdog+0x260/0x270Feb 10 07:01:54 pvehouston1 kernel: ? __pfx_dev_watchdog+0x10/0x10Feb 10 07:01:54 pvehouston1 kernel: call_timer_fn+0x29/0x160Feb 10 07:01:54 pvehouston1 kernel: ? __pfx_dev_watchdog+0x10/0x10Feb 10 07:01:54 pvehouston1 kernel: __run_timers+0x259/0x310Feb 10 07:01:54 pvehouston1 kernel: run_timer_softirq+0x1d/0x40Feb 10 07:01:54 pvehouston1 kernel: __do_softirq+0xd1/0x303Feb 10 07:01:54 pvehouston1 kernel: __irq_exit_rcu+0x75/0xa0Feb 10 07:01:54 pvehouston1 kernel: irq_exit_rcu+0xe/0x20Feb 10 07:01:54 pvehouston1 kernel: sysvec_apic_timer_interrupt+0x92/0xd0Feb 10 07:01:54 pvehouston1 kernel: </IRQ>Feb 10 07:01:54 pvehouston1 kernel: <TASK>Feb 10 07:01:54 pvehouston1 kernel: asm_sysvec_apic_timer_interrupt+0x1b/0x20Feb 10 07:01:54 pvehouston1 kernel: RIP: 0010
oll_idle+0x5d/0xb5Feb 10 07:01:54 pvehouston1 kernel: Code: 4f 02 20 49 8b 07 a8 08 75 32 4c 89 ef 48 89 de e8 38 ff ff ff 49 89 c5 b8 c9 00 00 00 49 8b 17 83 e2 08 75 17 f3 90 83 e8 01 <75> f1 e8 dc de ff ff 4c 29 e0 49 39 c5 73 df 80 0b 04 fa 0f 1f 44Feb 10 07:01:54 pvehouston1 kernel: RSP: 0018:ffffb637401efe18 EFLAGS: 00000206Feb 10 07:01:54 pvehouston1 kernel: RAX: 0000000000000082 RBX: ffffa01388b59400 RCX: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: RBP: ffffb637401efe40 R08: 0000000000000000 R09: 0000000000000000Feb 10 07:01:54 pvehouston1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00008a6bfb508992Feb 10 07:01:54 pvehouston1 kernel: R13: 0000000000008ca0 R14: 0000000000000000 R15: ffffa013808bc000Feb 10 07:01:54 pvehouston1 kernel: ? poll_idle+0x64/0xb5Feb 10 07:01:54 pvehouston1 kernel: cpuidle_enter_state+0x82/0x470Feb 10 07:01:54 pvehouston1 kernel: cpuidle_enter+0x2e/0x50Feb 10 07:01:54 pvehouston1 kernel: call_cpuidle+0x23/0x60Feb 10 07:01:54 pvehouston1 kernel: do_idle+0x202/0x260Feb 10 07:01:54 pvehouston1 kernel: cpu_startup_entry+0x2a/0x30Feb 10 07:01:54 pvehouston1 kernel: start_secondary+0x119/0x140Feb 10 07:01:54 pvehouston1 kernel: secondary_startup_64_no_verify+0x17e/0x18bFeb 10 07:01:54 pvehouston1 kernel: </TASK>Feb 10 07:01:54 pvehouston1 kernel: ---[ end trace 0000000000000000 ]---I tried updating my network driver, checking my network hardware and investigating my network traffic to see if there's a congestion issue. What should I be doing? Where should I be looking at?