kernel BUG at lib/dynamic_queue_limits.c:27!

[John_Doe]

Hi there,

had two hard crashes in the span of two days. I suspect a higher IO load as the trigger, although I can't be sure. system is up to date, running kernel 6.2.16-4-pve

Jul 20 20:07:19 server kernel: ------------[ cut here ]------------
Jul 20 20:07:19 server kernel: NETDEV WATCHDOG: enp6s0 (igc): transmit queue 2 timed out
Jul 20 20:07:19 server kernel: WARNING: CPU: 11 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Jul 20 20:07:19 server kernel: Modules linked in: tcp_diag inet_diag ipt_REJECT nf_reject_ipv4 nft_chain_nat nft_compat nf_conntrack_netlink xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE xfr>
Jul 20 20:07:19 server kernel:  ledtrig_audio intel_rapl_msr nouveau snd_pcm_dmaengine snd_hda_codec_hdmi intel_rapl_common x86_pkg_temp_thermal intel_powerclamp drm_ttm_helper snd_hda_int>
Jul 20 20:07:19 server kernel:  polyval_generic ghash_clmulni_intel sha512_ssse3 aesni_intel xhci_pci crypto_simd nvme spi_intel_pci xhci_pci_renesas cryptd i2c_i801 hpsa ahci e1000e spi_i>
Jul 20 20:07:19 server kernel: CPU: 11 PID: 0 Comm: swapper/11 Tainted: P           O       6.2.16-4-pve #1
Jul 20 20:07:19 server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7D09/Z590-A PRO (MS-7D09), BIOS 1.80 09/29/2022
Jul 20 20:07:19 server kernel: RIP: 0010:dev_watchdog+0x23a/0x250
Jul 20 20:07:19 server kernel: Code: 00 e9 2b ff ff ff 48 89 df c6 05 4a 6c 7d 01 01 e8 6b 08 f8 ff 44 89 f1 48 89 de 48 c7 c7 98 64 80 9b 48 89 c2 e8 86 a6 30 ff <0f> 0b e9 1c ff ff ff 66>
Jul 20 20:07:19 server kernel: RSP: 0018:ffffa387403d0e38 EFLAGS: 00010246
Jul 20 20:07:19 server kernel: RAX: 0000000000000000 RBX: ffff8e3693b6c000 RCX: 0000000000000000
Jul 20 20:07:19 server kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Jul 20 20:07:19 server kernel: RBP: ffffa387403d0e68 R08: 0000000000000000 R09: 0000000000000000
Jul 20 20:07:19 server kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8e3693b6c4c8
Jul 20 20:07:19 server kernel: R13: ffff8e3693b6c41c R14: 0000000000000002 R15: 0000000000000000
Jul 20 20:07:19 server kernel: FS:  0000000000000000(0000) GS:ffff8e45bfcc0000(0000) knlGS:0000000000000000
Jul 20 20:07:19 server kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 20 20:07:19 server kernel: CR2: 00007fa8d0036a90 CR3: 000000030f89e006 CR4: 0000000000772ee0
Jul 20 20:07:19 server kernel: PKRU: 55555554
Jul 20 20:07:19 server kernel: Call Trace:
Jul 20 20:07:19 server kernel:  <IRQ>
Jul 20 20:07:19 server kernel:  ? __pfx_dev_watchdog+0x10/0x10
Jul 20 20:07:19 server kernel:  call_timer_fn+0x29/0x160
Jul 20 20:07:19 server kernel:  ? __pfx_dev_watchdog+0x10/0x10
Jul 20 20:07:19 server kernel:  __run_timers+0x259/0x310
Jul 20 20:07:19 server kernel:  run_timer_softirq+0x1d/0x40
Jul 20 20:07:19 server kernel:  __do_softirq+0xd6/0x346
Jul 20 20:07:19 server kernel:  ? hrtimer_interrupt+0x11f/0x250
Jul 20 20:07:19 server kernel:  __irq_exit_rcu+0xa2/0xd0
Jul 20 20:07:19 server kernel:  irq_exit_rcu+0xe/0x20
Jul 20 20:07:19 server kernel:  sysvec_apic_timer_interrupt+0x92/0xd0
Jul 20 20:07:19 server kernel:  </IRQ>
Jul 20 20:07:19 server kernel:  <TASK>
Jul 20 20:07:19 server kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jul 20 20:07:19 server kernel: RIP: 0010:cpuidle_enter_state+0xde/0x6f0
Jul 20 20:07:19 server kernel: Code: 27 57 65 e8 d4 79 4a ff 8b 53 04 49 89 c7 0f 1f 44 00 00 31 ff e8 02 82 49 ff 80 7d d0 00 0f 85 eb 00 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 02 00>
Jul 20 20:07:19 server kernel: RSP: 0018:ffffa3874018be38 EFLAGS: 00000246
Jul 20 20:07:19 server kernel: RAX: 0000000000000000 RBX: ffffc3873fcc0300 RCX: 0000000000000000
Jul 20 20:07:19 server kernel: RDX: 000000000000000b RSI: 0000000000000000 RDI: 0000000000000000
Jul 20 20:07:19 server kernel: RBP: ffffa3874018be88 R08: 0000000000000000 R09: 0000000000000000
Jul 20 20:07:19 server kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9c2c33a0
Jul 20 20:07:19 server kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 000020cf14dd4384
Jul 20 20:07:19 server kernel:  ? cpuidle_enter_state+0xce/0x6f0
Jul 20 20:07:19 server kernel:  cpuidle_enter+0x2e/0x50
Jul 20 20:07:19 server kernel:  do_idle+0x216/0x2a0
Jul 20 20:07:19 server kernel:  cpu_startup_entry+0x1d/0x20
Jul 20 20:07:19 server kernel:  start_secondary+0x122/0x160
Jul 20 20:07:19 server kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Jul 20 20:07:19 server kernel:  </TASK>
Jul 20 20:07:19 server kernel: ---[ end trace 0000000000000000 ]---
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: Register Dump
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: Register Name   Value
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: CTRL            081c0641
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: STATUS          40380683
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: CTRL_EXT        10000040
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: MDIC            180a3800
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: ICR             000000c1
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: RCTL            0440803a
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: RDH[0-3]        000000e0 000000e2 0000005a 00000094
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: RDT[0-3]        000000df 000000e1 00000058 00000093
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: RDBAL[0-3]      0fa78000 17940000 1511c000 19fc6000
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: TCTL            a503f0fa
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: TDBAL[0-3]      18f7d000 0fbaa000 19b99000 0cc5d000
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: TDH[0-3]        00000059 00000002 0000003f 000000df
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: TDT[0-3]        00000059 00000002 0000003f 000000df
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Jul 20 20:07:19 server kernel: igc 0000:06:00.0 enp6s0: Reset adapter
Jul 20 20:07:20 server kernel: vmbr0: port 1(enp6s0) entered disabled state
Jul 20 20:07:20 server kernel: vmbr0v20: port 1(enp6s0.20) entered disabled state
Jul 20 20:07:20 server kernel: vmbr0v50: port 1(enp6s0.50) entered disabled state
Jul 20 20:07:20 server kernel: vmbr0v1000: port 1(enp6s0.1000) entered disabled state
Jul 20 20:07:23 server kernel: igc 0000:06:00.0 enp6s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jul 20 20:07:23 server kernel: vmbr0: port 1(enp6s0) entered blocking state
Jul 20 20:07:23 server kernel: vmbr0: port 1(enp6s0) entered forwarding state
Jul 20 20:07:23 server kernel: vmbr0v20: port 1(enp6s0.20) entered blocking state
Jul 20 20:07:23 server kernel: vmbr0v20: port 1(enp6s0.20) entered forwarding state
Jul 20 20:07:23 server kernel: vmbr0v50: port 1(enp6s0.50) entered blocking state
Jul 20 20:07:23 server kernel: vmbr0v50: port 1(enp6s0.50) entered forwarding state
Jul 20 20:07:23 server kernel: vmbr0v1000: port 1(enp6s0.1000) entered blocking state
Jul 20 20:07:23 server kernel: vmbr0v1000: port 1(enp6s0.1000) entered forwarding state
Jul 20 20:07:27 server kernel: ------------[ cut here ]------------
Jul 20 20:07:27 server kernel: kernel BUG at lib/dynamic_queue_limits.c:27!

I found this thread which seems to suggest the problem hasn't been fixed.

https://lore.kernel.org/lkml/6c389fde-4c8d-300b-8c3c-300d6105c30a@eikelenboom.it/T/

Seems like nothing can be done about it for the moment.....

 

[WidePeepoHappy]

Yeah, I just got this running Proxmox 8.0.3. Have a pfsense vm and a Ubuntu vm running docker with sabnzbd.

I don't know if it's ram or a software bug. System up rime before the crash was about 3 and half days.

Edit:
Here is my kernel panic log:

Jul 23 07:21:42 pve kernel: ------------[ cut here ]------------
Jul 23 07:21:42 pve kernel: NETDEV WATCHDOG: enp1s0 (igc): transmit queue 3 timed out
Jul 23 07:21:42 pve kernel: WARNING: CPU: 11 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Jul 23 07:21:42 pve kernel: Modules linked in: cls_u32 tcp_diag inet_diag act_police cls_basic sch_ingress sch_htb veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfnetlink_acct wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nf_tables sunrpc bonding tls softdog binfmt_misc nfnetlink_log nfnetlink snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_sof_pci_intel_cnl snd_sof_intel_hda_common intel_rapl_msr soundwire_intel intel_rapl_common soundwire_generic_allocation x86_pkg_temp_thermal soundwire_cadence intel_powerclamp snd_sof_intel_hda coretemp snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils kvm_intel snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus i915 kvm snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine
Jul 23 07:21:42 pve kernel:  snd_hda_intel crct10dif_pclmul drm_buddy polyval_clmulni polyval_generic snd_intel_dspcfg video snd_intel_sdw_acpi ghash_clmulni_intel wmi sha512_ssse3 ttm aesni_intel snd_hda_codec drm_display_helper crypto_simd snd_hda_core cryptd cec snd_hwdep cmdlinepart snd_pcm rc_core spi_nor snd_timer mtd rapl drm_kms_helper ee1004 snd intel_cstate soundcore i2c_algo_bit pcspkr syscopyarea sysfillrect sysimgblt intel_pch_thermal mac_hid vhost_net vhost vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb mmc_block nvme xhci_pci xhci_pci_renesas nvme_core sdhci_pci i2c_i801 spi_intel_pci crc32_pclmul nvme_common spi_intel i2c_smbus cqhci ahci igc xhci_hcd sdhci libahci pinctrl_cannonlake
Jul 23 07:21:42 pve kernel: CPU: 11 PID: 0 Comm: swapper/11 Tainted: P           O       6.2.16-4-pve #1
Jul 23 07:21:42 pve kernel: Hardware name: Protectli VP4670/VP4670, BIOS Dasharo (coreboot+UEFI) v1.1.0 04/14/2023
Jul 23 07:21:42 pve kernel: RIP: 0010:dev_watchdog+0x23a/0x250
Jul 23 07:21:42 pve kernel: Code: 00 e9 2b ff ff ff 48 89 df c6 05 4a 6c 7d 01 01 e8 6b 08 f8 ff 44 89 f1 48 89 de 48 c7 c7 98 64 80 86 48 89 c2 e8 86 a6 30 ff <0f> 0b e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00
Jul 23 07:21:42 pve kernel: RSP: 0018:ffffb66940350e38 EFLAGS: 00010246
Jul 23 07:21:42 pve kernel: RAX: 0000000000000000 RBX: ffffa015c20c6000 RCX: 0000000000000000
Jul 23 07:21:42 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Jul 23 07:21:42 pve kernel: RBP: ffffb66940350e68 R08: 0000000000000000 R09: 0000000000000000
Jul 23 07:21:42 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa015c20c64c8
Jul 23 07:21:42 pve kernel: R13: ffffa015c20c641c R14: 0000000000000003 R15: 0000000000000000
Jul 23 07:21:42 pve kernel: FS:  0000000000000000(0000) GS:ffffa024de4c0000(0000) knlGS:0000000000000000
Jul 23 07:21:42 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 23 07:21:42 pve kernel: CR2: 00007fc4d2c76b98 CR3: 0000000178170004 CR4: 00000000003726e0
Jul 23 07:21:42 pve kernel: Call Trace:
Jul 23 07:21:42 pve kernel:  <IRQ>
Jul 23 07:21:42 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Jul 23 07:21:42 pve kernel:  call_timer_fn+0x29/0x160
Jul 23 07:21:42 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Jul 23 07:21:42 pve kernel:  __run_timers+0x259/0x310
Jul 23 07:21:42 pve kernel:  run_timer_softirq+0x1d/0x40
Jul 23 07:21:42 pve kernel:  __do_softirq+0xd6/0x346
Jul 23 07:21:42 pve kernel:  ? hrtimer_interrupt+0x11f/0x250
Jul 23 07:21:42 pve kernel:  __irq_exit_rcu+0xa2/0xd0
Jul 23 07:21:42 pve kernel:  irq_exit_rcu+0xe/0x20
Jul 23 07:21:42 pve kernel:  sysvec_apic_timer_interrupt+0x92/0xd0
Jul 23 07:21:42 pve kernel:  </IRQ>
Jul 23 07:21:42 pve kernel:  <TASK>
Jul 23 07:21:42 pve kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jul 23 07:21:42 pve kernel: RIP: 0010:cpuidle_enter_state+0xde/0x6f0
Jul 23 07:21:42 pve kernel: Code: 27 57 7a e8 d4 79 4a ff 8b 53 04 49 89 c7 0f 1f 44 00 00 31 ff e8 02 82 49 ff 80 7d d0 00 0f 85 eb 00 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 02 00 00 4d 63 ee 49 83 fd 09 0f 87 c7 04 00 00
Jul 23 07:21:42 pve kernel: RSP: 0018:ffffb6694010be38 EFLAGS: 00000246
Jul 23 07:21:42 pve kernel: RAX: 0000000000000000 RBX: ffffd6693fcc0200 RCX: 0000000000000000
Jul 23 07:21:42 pve kernel: RDX: 000000000000000b RSI: 0000000000000000 RDI: 0000000000000000
Jul 23 07:21:42 pve kernel: RBP: ffffb6694010be88 R08: 0000000000000000 R09: 0000000000000000
Jul 23 07:21:42 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff872c33a0
Jul 23 07:21:42 pve kernel: R13: 0000000000000001 R14: 0000000000000001 R15: 00012488064e81bc
Jul 23 07:21:42 pve kernel:  ? cpuidle_enter_state+0xce/0x6f0
Jul 23 07:21:42 pve kernel:  cpuidle_enter+0x2e/0x50
Jul 23 07:21:42 pve kernel:  do_idle+0x216/0x2a0
Jul 23 07:21:42 pve kernel:  cpu_startup_entry+0x1d/0x20
Jul 23 07:21:42 pve kernel:  start_secondary+0x122/0x160
Jul 23 07:21:42 pve kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Jul 23 07:21:42 pve kernel:  </TASK>
Jul 23 07:21:42 pve kernel: ---[ end trace 0000000000000000 ]---
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: Register Dump
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: Register Name   Value
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: CTRL            081c0641
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: STATUS          40680683
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: CTRL_EXT        10000040
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: MDIC            1805d181
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: ICR             000000c1
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: RCTL            0440803a
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: RDH[0-3]        000000fb 000000d3 0000005c 000000fd
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: RDT[0-3]        000000fa 000000d2 00000054 000000fc
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: RDBAL[0-3]      0db39000 08820000 06553000 0c9d5000
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: TCTL            a503f0fa
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: TDBAL[0-3]      11e92000 11ea7000 0dc20000 0bb54000
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: TDH[0-3]        000000a1 0000000b 000000ae 00000073
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: TDT[0-3]        000000a1 0000000d 000000ae 00000073
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Jul 23 07:21:42 pve kernel: igc 0000:01:00.0 enp1s0: Reset adapter
Jul 23 07:21:42 pve kernel: vmbr0: port 1(enp1s0) entered disabled state
Jul 23 07:21:46 pve kernel: igc 0000:01:00.0 enp1s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jul 23 07:21:46 pve kernel: vmbr0: port 1(enp1s0) entered blocking state
Jul 23 07:21:46 pve kernel: vmbr0: port 1(enp1s0) entered forwarding state
Jul 23 07:21:50 pve kernel: ------------[ cut here ]------------
Jul 23 07:21:50 pve kernel: refcount_t: underflow; use-after-free.
Jul 23 07:21:50 pve kernel: WARNING: CPU: 1 PID: 0 at lib/refcount.c:28 refcount_warn_saturate+0xa3/0x150
Jul 23 07:21:50 pve kernel: Modules linked in: cls_u32 tcp_diag inet_diag act_police cls_basic sch_ingress sch_htb veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfnetlink_acct wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nf_tables sunrpc bonding tls softdog binfmt_misc nfnetlink_log nfnetlink snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_sof_pci_intel_cnl snd_sof_intel_hda_common intel_rapl_msr soundwire_intel intel_rapl_common soundwire_generic_allocation x86_pkg_temp_thermal soundwire_cadence intel_powerclamp snd_sof_intel_hda coretemp snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils kvm_intel snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus i915 kvm snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine
Jul 23 07:21:50 pve kernel:  snd_hda_intel crct10dif_pclmul drm_buddy polyval_clmulni polyval_generic snd_intel_dspcfg video snd_intel_sdw_acpi ghash_clmulni_intel wmi sha512_ssse3 ttm aesni_intel snd_hda_codec drm_display_helper crypto_simd snd_hda_core cryptd cec snd_hwdep cmdlinepart snd_pcm rc_core spi_nor snd_timer mtd rapl drm_kms_helper ee1004 snd intel_cstate soundcore i2c_algo_bit pcspkr syscopyarea sysfillrect sysimgblt intel_pch_thermal mac_hid vhost_net vhost vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb mmc_block nvme xhci_pci xhci_pci_renesas nvme_core sdhci_pci i2c_i801 spi_intel_pci crc32_pclmul nvme_common spi_intel i2c_smbus cqhci ahci igc xhci_hcd sdhci libahci pinctrl_cannonlake
Jul 23 07:21:50 pve kernel: CPU: 1 PID: 0 Comm: swapper/1 Tainted: P        W  O       6.2.16-4-pve #1
Jul 23 07:21:50 pve kernel: Hardware name: Protectli VP4670/VP4670, BIOS Dasharo (coreboot+UEFI) v1.1.0 04/14/2023
Jul 23 07:21:50 pve kernel: RIP: 0010:refcount_warn_saturate+0xa3/0x150
Jul 23 07:21:50 pve kernel: Code: cc cc 0f b6 1d e0 7b e0 01 80 fb 01 0f 87 99 98 88 00 83 e3 01 75 dd 48 c7 c7 48 e5 76 86 c6 05 c4 7b e0 01 01 e8 1d b7 93 ff <0f> 0b eb c6 0f b6 1d b7 7b e0 01 80 fb 01 0f 87 59 98 88 00 83 e3
Jul 23 07:21:50 pve kernel: RSP: 0018:ffffb66940148d30 EFLAGS: 00010246
Jul 23 07:21:50 pve kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Jul 23 07:21:50 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Jul 23 07:21:50 pve kernel: RBP: ffffb66940148d38 R08: 0000000000000000 R09: 0000000000000000
Jul 23 07:21:50 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Jul 23 07:21:50 pve kernel: R13: ffffa015d6709154 R14: ffffa015cbb54c80 R15: 00000000ffffffb7
Jul 23 07:21:50 pve kernel: FS:  0000000000000000(0000) GS:ffffa024de240000(0000) knlGS:0000000000000000
Jul 23 07:21:50 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 23 07:21:50 pve kernel: CR2: 000000c000dba0c8 CR3: 0000000178170004 CR4: 00000000003726e0
Jul 23 07:21:50 pve kernel: Call Trace:
Jul 23 07:21:50 pve kernel:  <IRQ>
Jul 23 07:21:50 pve kernel:  sock_wfree+0x118/0x200
Jul 23 07:21:50 pve kernel:  skb_release_head_state+0x24/0xb0
Jul 23 07:21:50 pve kernel:  napi_consume_skb+0x3f/0x180
Jul 23 07:21:50 pve kernel:  igc_poll+0x874/0x17d0 [igc]
Jul 23 07:21:50 pve kernel:  ? timekeeping_update+0xd0/0x160
Jul 23 07:21:50 pve kernel:  ? ring_buffer_lock_reserve+0x14c/0x3e0
Jul 23 07:21:50 pve kernel:  ? trace_call_bpf+0xd5/0x160
Jul 23 07:21:50 pve kernel:  __napi_poll+0x30/0x1f0
Jul 23 07:21:50 pve kernel:  net_rx_action+0x180/0x2d0
Jul 23 07:21:50 pve kernel:  __do_softirq+0xd6/0x346
Jul 23 07:21:50 pve kernel:  ? handle_irq_event+0x52/0x80
Jul 23 07:21:50 pve kernel:  ? handle_edge_irq+0xda/0x250
Jul 23 07:21:50 pve kernel:  __irq_exit_rcu+0xa2/0xd0
Jul 23 07:21:50 pve kernel:  irq_exit_rcu+0xe/0x20
Jul 23 07:21:50 pve kernel:  common_interrupt+0xa4/0xb0
Jul 23 07:21:50 pve kernel:  </IRQ>
Jul 23 07:21:50 pve kernel:  <TASK>
Jul 23 07:21:50 pve kernel:  asm_common_interrupt+0x27/0x40
Jul 23 07:21:50 pve kernel: RIP: 0010:cpuidle_enter_state+0xde/0x6f0
Jul 23 07:21:50 pve kernel: Code: 27 57 7a e8 d4 79 4a ff 8b 53 04 49 89 c7 0f 1f 44 00 00 31 ff e8 02 82 49 ff 80 7d d0 00 0f 85 eb 00 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 02 00 00 4d 63 ee 49 83 fd 09 0f 87 c7 04 00 00
Jul 23 07:21:50 pve kernel: RSP: 0018:ffffb669400bbe38 EFLAGS: 00000246
Jul 23 07:21:50 pve kernel: RAX: 0000000000000000 RBX: ffffd6693fa40200 RCX: 0000000000000000
Jul 23 07:21:50 pve kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000
Jul 23 07:21:50 pve kernel: RBP: ffffb669400bbe88 R08: 0000000000000000 R09: 0000000000000000
Jul 23 07:21:50 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff872c33a0
Jul 23 07:21:50 pve kernel: R13: 0000000000000001 R14: 0000000000000001 R15: 00012489f72d1033
Jul 23 07:21:50 pve kernel:  ? cpuidle_enter_state+0xce/0x6f0
Jul 23 07:21:50 pve kernel:  cpuidle_enter+0x2e/0x50
Jul 23 07:21:50 pve kernel:  do_idle+0x216/0x2a0
Jul 23 07:21:50 pve kernel:  cpu_startup_entry+0x1d/0x20
Jul 23 07:21:50 pve kernel:  start_secondary+0x122/0x160
Jul 23 07:21:50 pve kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Jul 23 07:21:50 pve kernel:  </TASK>
Jul 23 07:21:50 pve kernel: ---[ end trace 0000000000000000 ]---
Jul 23 07:21:50 pve kernel: ------------[ cut here ]------------
Jul 23 07:21:50 pve kernel: kernel BUG at lib/dynamic_queue_limits.c:27!
-- Reboot --

 

[fweber]

Hello,

seems like you both use the Intel igc networking driver:

Jul 20 20:07:19 server kernel: NETDEV WATCHDOG: enp6s0 (igc): transmit queue 2 timed out

Jul 23 07:21:42 pve kernel: NETDEV WATCHDOG: enp1s0 (igc): transmit queue 3 timed out

Can I ask which NIC this is -- could you post the output of lspci -nn | grep Ethernet?

There is a recent patch on a kernel mailing list that fixes a bug that produces a very similar backtrace [1]. AFAICT, the patch has not made its way into the kernel yet. However, if I read the patch right, the bug was apparently introduced in an earlier commit [2], which is included in PVE kernel >= 6.2.9 (and, in the 5.15 line, >= 5.15.102).

To find out if it is really the commit [2] causing the problems, you could try to temporarily downgrade the kernel to a version without the commit [2], and see if the situation improves. You could try, for example, apt install pve-kernel-6.1, and then pin the 6.1 kernel for the next boot using proxmox-boot-tool [3]. After the reboot, you can check that you are indeed running a 6.1 kernel by checking the output of uname -a.

It would be great if you could report back whether the problems persist or not. Note that even if the problems should disappear, staying at kernel 6.1 would not be a permanent solution, as 6.1 is not supported in PVE 8.

[1] https://lore.kernel.org/intel-wired-lan/c5c4501c-5719-6a62-5012-91e34b5d7dcc@intel.com/T/
[2] https://git.kernel.org/pub/scm/linu...3&id=9b275176270efd18f2f4e328b32be1bad34c4c0d
[3] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_kernel_pin

 

[WidePeepoHappy]

Can I ask which NIC this is -- could you post the output of lspci -nn | grep Ethernet?

01:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) 02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) 03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) 04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) 05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) 06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)

I am still waiting for a second crash to see if it's the same error. After that, I'll switch to kernel 6.1. As you said kernel 6.1 is not supported in PVE 8 it has me worried that switching to it will break something else. As I run the PVE host at my brother's place with a pfsense VM if something goes wrong his internet is gone, with him not knowing how to fix it.

 

[fweber]

Thanks for the information! Apparently the I225-V occasionally causes some problems under Linux, e.g. [1] reports connection drops (the issue you report seems different, though).

Note that kernel 6.1 being unsupported means that we do not ship any updates for the 6.1 line and do not help with issues specific to the 6.1 line -- it does not mean that PVE is inherently incompatible with a 6.1 kernel. But of course, something else breaking with a 6.1 kernel cannot be completely ruled out. So I can understand that trying the 6.1 kernel seems risky if you don't have direct access to the PVE host. If it helps, you can just pin the 6.1 kernel for the next boot [2], and boot back into the 6.2 kernel if anything seems off.

You can also wait a few days and see what happens with the kernel patch I linked. It does seem to be actively worked on, see yesterday's post on the Linux netdev list [3]. However, note that even if this patch gets applied, it can take a few weeks/months until it is incorporated into a PVE kernel.

[1] https://www.reddit.com/r/buildapc/comments/xypn1m/network_card_intel_ethernet_controller_i225v_igc/
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_kernel_pin
[3] https://lore.kernel.org/netdev/20230724161250.2177321-1-anthony.l.nguyen@intel.com/T/

 

[Huamijo]

came here from https://forum.proxmox.com/threads/u...ransmit-queue-0-timed-out.130415/#post-580211

Pinning 6.1.10-1-pve "worked" now i don't get any host freezes anymore, but a lot of Tx Unit Hang messages.
It just took 4 more hours than usual and i get the same result plus the added bonus of a full syslog with this msg every 2 seconds.

Aug 10 10:55:39 XXXXX kernel: igc 0000:06:00.0 enp6s0: Detected Tx Unit Hang
  Tx Queue             <3>
  TDH                  <a>
  TDT                  <a>
  next_to_use          <e4>
  next_to_clean        <a>
buffer_info[next_to_clean]
  time_stamp           <1000138dc>
  next_to_watch        <0000000058fef5f8>
  jiffies              <100015988>
  desc.status          <0>
Aug 10 10:55:41 XXXXX kernel: igc 0000:06:00.0 enp6s0: Detected Tx Unit Hang
  Tx Queue             <3>
  TDH                  <a>
  TDT                  <a>
  next_to_use          <e4>
  next_to_clean        <a>
buffer_info[next_to_clean]
  time_stamp           <1000138dc>
  next_to_watch        <0000000058fef5f8>
  jiffies              <100015b79>
  desc.status          <0>
Aug 10 10:55:43 XXXXX kernel: igc 0000:06:00.0 enp6s0: Detected Tx Unit Hang
  Tx Queue             <3>
  TDH                  <a>
  TDT                  <a>
  next_to_use          <e4>
  next_to_clean        <a>
buffer_info[next_to_clean]
  time_stamp           <1000138dc>
  next_to_watch        <0000000058fef5f8>
  jiffies              <100015d70>
  desc.status          <0>

for now this is acceptable, i preffer a log dumpster over a frozen host if i'm not near the hw.
As this was a new side project, i will shut down proxmox ve for now and wait until there is a patched linux/pve kernel.

Hopefully the patch -> new kernel -> new pve-kernel will be here soon

 

[Huamijo]

had to update my comment, @John_Doe you might want to wait until you try it yourself

 

[fweber]

Thanks for trying the 6.1 kernel and reporting back your results! Sorry to hear it didn't help much. The "Detected Tx Unit Hang" might be the false positives mentioned in the commit message of [1] -- kernel 6.1 does not have this commit, so it would make sense that the log messages started to appear with kernel 6.1. Do I understand correctly that you also the NETDEV WATCHDOG: enp6s0 (igc): transmit queue 0 timed out message after ~4 hours?

We tried to reproduce the issue on our end with an Intel I225-V, but didn't have any luck so far.

[1] https://git.kernel.org/pub/scm/linu...3&id=9b275176270efd18f2f4e328b32be1bad34c4c0d

 

[Huamijo]

Thank you for testing it yourself with i225-v.
For some reason i didn't notice this thread only talked about i225-v. I run a box with i226-v, so i will wait for the i225-v fix and hope this fixes i226-v too.

:~# lspci -nn | grep Ethernet
02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)

Those were the last log entries.

kernel: igc 0000:06:00.0 enp6s0: Detected Tx Unit Hang
  Tx Queue             <0>
  TDH                  <a7>
  TDT                  <a7>
  next_to_use          <8e>
  next_to_clean        <a7>
buffer_info[next_to_clean]
  time_stamp           <1002921b8>
  next_to_watch        <000000003e292517>
  jiffies              <10029dda0>
  desc.status          <0>
kernel: igc 0000:06:00.0 enp6s0: NIC Link is Down
kernel: vmbr1: port 4(enp6s0) entered disabled state
kernel: igc 0000:06:00.0 enp6s0: Register Dump
kernel: igc 0000:06:00.0 enp6s0: Register Name   Value
kernel: igc 0000:06:00.0 enp6s0: CTRL            181c0641
kernel: igc 0000:06:00.0 enp6s0: STATUS          00680681
kernel: igc 0000:06:00.0 enp6s0: CTRL_EXT        100000c0
kernel: igc 0000:06:00.0 enp6s0: MDIC            18017949
kernel: igc 0000:06:00.0 enp6s0: ICR             00000000
kernel: igc 0000:06:00.0 enp6s0: RCTL            0440803a
kernel: igc 0000:06:00.0 enp6s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
kernel: igc 0000:06:00.0 enp6s0: RDH[0-3]        000000f4 000000f4 000000bd 0000008f
kernel: igc 0000:06:00.0 enp6s0: RDT[0-3]        000000f3 000000f3 000000bc 0000008e
kernel: igc 0000:06:00.0 enp6s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
kernel: igc 0000:06:00.0 enp6s0: RDBAL[0-3]      11fb2000 3a3ae000 11fbe000 11fba000
kernel: igc 0000:06:00.0 enp6s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
kernel: igc 0000:06:00.0 enp6s0: TCTL            a503f0fa
kernel: igc 0000:06:00.0 enp6s0: TDBAL[0-3]      11faa000 0586a000 1da9f000 11fb5000
kernel: igc 0000:06:00.0 enp6s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
kernel: igc 0000:06:00.0 enp6s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
kernel: igc 0000:06:00.0 enp6s0: TDH[0-3]        000000a7 0000001b 000000fe 00000044
kernel: igc 0000:06:00.0 enp6s0: TDT[0-3]        000000a7 0000001b 000000fe 00000044
kernel: igc 0000:06:00.0 enp6s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
kernel: igc 0000:06:00.0 enp6s0: Reset adapter
-- Reboot --
kernel: Linux version 6.2.16-6-pve (fgruenbichler@yuna) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-7 (2023-08-01T11:23Z) ()
kernel: Command line: initrd=\EFI\proxmox\6.2.16-6-pve\initrd.img-6.2.16-6-pve root=ZFS=rpool/ROOT/XXX boot=zfs cpufreq.default_governor=powersave pcie_aspm=off pcie_port_pm=off

After the freeze and hard reset it booted with the default kernel. I only pinned 6.1 for the next boot, bc i didn't know how stable i226-v is in 6.1.

 

[tictag]

Further discussion on a similar issue (different behavioural cause and NIC, but same error) here. Not sure if threads should be merged?

 

[fweber]

Hi, the patch I mentioned earlier [1] has now been backported to our PVE kernel [2] and is available in proxmox-kernel >= 6.2.16-9. So you could try installing and booting into the latest 6.2 kernel to see if it fixes the kernel bugs and freezes. If you do, it would be great if you could report back your results.

[1] https://lore.kernel.org/intel-wired-lan/c5c4501c-5719-6a62-5012-91e34b5d7dcc@intel.com/T/
[2] https://git.proxmox.com/?p=pve-kernel.git;a=commit;h=8b9dc0218075f856d0a7a8d3120d515905b1d6b9

 

[tictag]

Hi, the patch I mentioned earlier [1] has now been backported to our PVE kernel [2] and is available in proxmox-kernel >= 6.2.16-9. So you could try installing and booting into the latest 6.2 kernel to see if it fixes the kernel bugs and freezes. If you do, it would be great if you could report back your results.

I will be trying this tomorrow. Spent today figuring out how to make a backup of Proxmox VE itself ... that was a fun journey. Sorted now.

 

[tictag]

Updated post #3 in thread Another kernel BUG at lib/dynamic_queue_limits.c:27!

Note: not sure if these two threads should be merged ... same outcome but different causes.

 

[zack0ne]

Fresh installed Proxmox v8 yesterday from v7. I also encounter similar problem with Dell Optiplex 3070 mini. Usually not responsive after 20-30minute after boot (had to hard reset).

Things i've configured:

1. Add SAMBA datacenter storage (backup from v7)
2. Restore VM/LXC from SAMBA backup.
3. Configure iGPU for LXC hardware encoder.

Things i've tried to solve the issue:

1. Update packages to latest version.
2. Remove Mapped PCI Intel HD.
3. Remove Mapped USB External HDD (uses qm set XXX --scsi now).
4. Remove, clean and reinsert 16+8GB RAM.
5. Remove SAMBA datacenter storage (uses for backup).
6. Remove case cover (maybe overheat??)

Sep 02 09:08:44 roboco kernel: ------------[ cut here ]------------
Sep 02 09:08:44 roboco kernel: NETDEV WATCHDOG: enp1s0 (r8169): transmit queue 0 timed out
Sep 02 09:08:44 roboco kernel: WARNING: CPU: 4 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Sep 02 09:08:44 roboco kernel: Modules linked in: cmac nls_utf8 cifs cifs_arc4 rdma_cm iw_cm ib_cm ib_core cifs_md4 fscache netfs tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc nfnetlink_log nfnetlink binfmt_misc snd_hda_codec_hdmi snd_ctl_led snd_hda_codec_realtek snd_hda_codec_generic snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus intel_rapl_msr intel_rapl_common intel_tcc_cooling x86_pkg_temp_thermal snd_soc_core intel_powerclamp coretemp snd_compress kvm_intel ac97_bus i915 snd_pcm_dmaengine snd_hda_intel kvm drm_buddy snd_intel_dspcfg mei_pxp mei_hdcp irqbypass ttm snd_intel_sdw_acpi drm_display_helper crct10dif_pclmul snd_hda_codec polyval_clmulni polyval_generic
Sep 02 09:08:44 roboco kernel:  snd_hda_core ghash_clmulni_intel sha512_ssse3 cec aesni_intel snd_hwdep rc_core snd_pcm dell_wmi drm_kms_helper ledtrig_audio crypto_simd snd_timer cryptd dell_wmi_sysman dell_smbios rapl intel_cstate firmware_attributes_class pcspkr cmdlinepart snd dcdbas i2c_algo_bit spi_nor mei_me syscopyarea sysfillrect ee1004 sparse_keymap dell_wmi_descriptor soundcore mtd wmi_bmof sysimgblt mei intel_pch_thermal acpi_pad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb uas usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme xhci_pci nvme_core r8169 spi_intel_pci xhci_pci_renesas crc32_pclmul xhci_hcd nvme_common i2c_i801 realtek ahci i2c_smbus spi_intel libahci video wmi
Sep 02 09:08:44 roboco kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: P           O       6.2.16-10-pve #1
Sep 02 09:08:44 roboco kernel: Hardware name: Dell Inc. OptiPlex 3070/05YDCW, BIOS 1.3.1 02/06/2020
Sep 02 09:08:44 roboco kernel: RIP: 0010:dev_watchdog+0x23a/0x250
Sep 02 09:08:44 roboco kernel: Code: 00 e9 2b ff ff ff 48 89 df c6 05 8c 68 7d 01 01 e8 6b 08 f8 ff 44 89 f1 48 89 de 48 c7 c7 b0 6e e0 ad 48 89 c2 e8 86 9f 30 ff <0f> 0b e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00
Sep 02 09:08:44 roboco kernel: RSP: 0018:ffffac02001f8e38 EFLAGS: 00010246
Sep 02 09:08:44 roboco kernel: RAX: 0000000000000000 RBX: ffff8f3746144000 RCX: 0000000000000000
Sep 02 09:08:44 roboco kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 02 09:08:44 roboco kernel: RBP: ffffac02001f8e68 R08: 0000000000000000 R09: 0000000000000000
Sep 02 09:08:44 roboco kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8f37461444c8
Sep 02 09:08:44 roboco kernel: R13: ffff8f374614441c R14: 0000000000000000 R15: 0000000000000000
Sep 02 09:08:44 roboco kernel: FS:  0000000000000000(0000) GS:ffff8f3c66500000(0000) knlGS:0000000000000000
Sep 02 09:08:44 roboco kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 02 09:08:44 roboco kernel: CR2: 00007f9277298ae0 CR3: 000000031fa10004 CR4: 00000000003726e0
Sep 02 09:08:44 roboco kernel: Call Trace:
Sep 02 09:08:44 roboco kernel:  <IRQ>
Sep 02 09:08:44 roboco kernel:  ? __pfx_dev_watchdog+0x10/0x10
Sep 02 09:08:44 roboco kernel:  call_timer_fn+0x29/0x160
Sep 02 09:08:44 roboco kernel:  ? __pfx_dev_watchdog+0x10/0x10
Sep 02 09:08:44 roboco kernel:  __run_timers+0x259/0x310
Sep 02 09:08:44 roboco kernel:  run_timer_softirq+0x1d/0x40
Sep 02 09:08:44 roboco kernel:  __do_softirq+0xd6/0x346
Sep 02 09:08:44 roboco kernel:  ? hrtimer_interrupt+0x11f/0x250
Sep 02 09:08:44 roboco kernel:  __irq_exit_rcu+0xa2/0xd0
Sep 02 09:08:44 roboco kernel:  irq_exit_rcu+0xe/0x20
Sep 02 09:08:44 roboco kernel:  sysvec_apic_timer_interrupt+0x92/0xd0
Sep 02 09:08:44 roboco kernel:  </IRQ>
Sep 02 09:08:44 roboco kernel:  <TASK>
Sep 02 09:08:44 roboco kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Sep 02 09:08:44 roboco kernel: RIP: 0010:cpuidle_enter_state+0xde/0x6f0
Sep 02 09:08:44 roboco kernel: Code: 20 f7 52 e8 d4 72 4a ff 8b 53 04 49 89 c7 0f 1f 44 00 00 31 ff e8 02 7b 49 ff 80 7d d0 00 0f 85 eb 00 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 02 00 00 4d 63 ee 49 83 fd 09 0f 87 c7 04 00 00
Sep 02 09:08:44 roboco kernel: RSP: 0018:ffffac020011be38 EFLAGS: 00000246
Sep 02 09:08:44 roboco kernel: RAX: 0000000000000000 RBX: ffffcc01ffd1f000 RCX: 0000000000000000
Sep 02 09:08:44 roboco kernel: RDX: 0000000000000004 RSI: 0000000000000000 RDI: 0000000000000000
Sep 02 09:08:44 roboco kernel: RBP: ffffac020011be88 R08: 0000000000000000 R09: 0000000000000000
Sep 02 09:08:44 roboco kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffae8c36c0
Sep 02 09:08:44 roboco kernel: R13: 0000000000000006 R14: 0000000000000006 R15: 000005f5897bd19a
Sep 02 09:08:44 roboco kernel:  ? cpuidle_enter_state+0xce/0x6f0
Sep 02 09:08:44 roboco kernel:  cpuidle_enter+0x2e/0x50
Sep 02 09:08:44 roboco kernel:  do_idle+0x216/0x2a0
Sep 02 09:08:44 roboco kernel:  cpu_startup_entry+0x1d/0x20
Sep 02 09:08:44 roboco kernel:  start_secondary+0x122/0x160
Sep 02 09:08:44 roboco kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Sep 02 09:08:44 roboco kernel:  </TASK>
Sep 02 09:08:44 roboco kernel: ---[ end trace 0000000000000000 ]---
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Sep 02 09:08:44 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Sep 02 09:11:02 roboco kernel: net_ratelimit: 9 callbacks suppressed

…. REMOVED LINES …

Sep 02 10:07:02 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Sep 02 10:07:02 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Sep 02 10:07:02 roboco kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).

…. MANUALLY POWERING DOWN BY PRESSING BUTTON …
Sep 02 10:08:49 roboco systemd-logind[710]: Power key pressed short.
Sep 02 10:08:49 roboco systemd-logind[710]: Powering off...
Sep 02 10:08:49 roboco systemd-logind[710]: System is powering down.

Hi, the patch I mentioned earlier [1] has now been backported to our PVE kernel [2] and is available in proxmox-kernel >= 6.2.16-9. So you could try installing and booting into the latest 6.2 kernel to see if it fixes the kernel bugs and freezes. If you do, it would be great if you could report back your results.

[1] https://lore.kernel.org/intel-wired-lan/c5c4501c-5719-6a62-5012-91e34b5d7dcc@intel.com/T/
[2] https://git.proxmox.com/?p=pve-kernel.git;a=commit;h=8b9dc0218075f856d0a7a8d3120d515905b1d6b9

Right now i've installed the mentioned kernal apt install proxmox-kernel-6.2.16-9-pve. After 1h40m uptime still responsive.

UPDATE: It happened again after 2h~ uptime. This time i've managed to connect monitor + keyboard to access directly. seem like all running well just without network. Then i perform the following:

1. Run Memtest86 v6.10, PASS no error
2. DISABLE sleep mode in BIOS.

Currently with 3h30m uptime and i didn't see any NETDEV WATCHDOG: enp1s0 (r8169): transmit queue 0 timed out.

UPDATE 2: After 2days uptime, seem fine, I don't think the number 2 above

• DISABLE sleep mode in BIOS.

would cause the problem since it's ENABLED while using proxmox v7 before.

I will keep monitoring and update here if it still happen.

Thank you, If require any more information I am happy to help.

Additional information might help

# spci -nn | grep Ethernet
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

proxmox-ve: 8.0.2 (running kernel: 6.2.16-10-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-10-pve: 6.2.16-10
proxmox-kernel-6.2: 6.2.16-10
proxmox-kernel-6.2.16-9-pve: 6.2.16-9
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.8
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-5
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

 

[fweber]

Hi @zack0ne, thanks for sharing your observations. Have you experienced another crash since you updated the Kernel to 6.2.16-10 and disabled sleep mode in BIOS? If you saw another crash, could you check the journal using journalctl and send the messages around the time of the crash (30min before until and including the crash)?

UPDATE 2: After 2days uptime, seem fine, I don't think the number 2 above

would cause the problem since it's ENABLED while using proxmox v7 before.

Which kernel did you use under PVE 7.4 -- was it a 5.15 kernel? Actually, I could imagine the power save making a difference here, for example if a bug was introduced some time between kernel 5.15 and kernel 6.2 that is more likely to trigger when sleep mode is enabled.

 

[zack0ne]

Hi @zack0ne, thanks for sharing your observations. Have you experienced another crash since you updated the Kernel to 6.2.16-10 and disabled sleep mode in BIOS? If you saw another crash, could you check the journal using journalctl and send the messages around the time of the crash (30min before until and including the crash)?

After updated kernal 6.2.16-10 and disabled sleep in bios I've never saw another crash. Attached here several crash journal log from booting to crashing (filtered by Boot ID).

Which kernel did you use under PVE 7.4 -- was it a 5.15 kernel? Actually, I could imagine the power save making a difference here, for example if a bug was introduced some time between kernel 5.15 and kernel 6.2 that is more likely to trigger when sleep mode is enabled.

I've check my previous install (luckily i haven't purge it). it's 5.15.30-2-pve kernal.

 

[fweber]

Hi @zack0ne, thanks. One thing that I overlooked earlier is that according to your logs, your NIC does not use the igc driver (like it was the case for the other posters in this thread), but the r8169 driver. So you're likely seeing a different bug than the other posters in this thread. See e.g. this thread [1] for more information on the r8169 driver hang -- apparently it can be fixed by installing the r8168-dkms package.

[1] https://forum.proxmox.com/threads/networking-issues-pve8.129742/post-569665

 

[Huamijo]

Got some good news.
With 6.2.16-12-pve there is no host freeze up and no errors displayed in syslog.
Tested with a 6h SMB file transfere transfere at >100MiB/s and a week of normal runtime.
Previously it would freez between 30min and 5h in the transfere or some random time a day.

Now it seems the problem with slow port negonation resulting in a couple of seconds for it to go up is worse. But if this is a pve problem, what i don't know right now, it calls for a new thread.


In my opinion this thread can me marked solved.
Solution: Update to at least kernel 6.2.16-12-pve.

 

[weeeeems]

Hi

My Proxmox server became totally unresponsive today in what seems like a related case to this thread. It is only being used for OPNsense and Home Assistant right now.

Proxmox 8, kernal 6.2.16-3-pve

root@pve:~# pveversion
pve-manager/8.0.3/bbf3993334bfa916 (running kernel: 6.2.16-3-pve)

Intel I226-V runnin igc driver

root@pve:~# lspci -nn | grep Ethernet
02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)

Log entry:

Sep 17 09:12:54 pve kernel: ------------[ cut here ]------------
Sep 17 09:12:54 pve kernel: NETDEV WATCHDOG: enp3s0 (igc): transmit queue 0 timed out
Sep 17 09:12:54 pve kernel: WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Sep 17 09:12:54 pve kernel: Modules linked in: cfg80211 veth tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc nfnetlink_log nfnetlink binfmt_misc snd_hda_codec_hdmi snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus intel_rapl_msr i915 intel_rapl_common snd_soc_core x86_pkg_temp_thermal intel_powerclamp snd_compress ac97_bus coretemp snd_pcm_dmaengine snd_hda_intel drm_buddy kvm_intel ttm snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm drm_display_helper irqbypass crct10dif_pclmul cec polyval_clmulni polyval_generic ghash_clmulni_intel snd_hda_core sha512_ssse3 aesni_intel rc_core crypto_simd snd_hwdep ov13858 cryptd v4l2_fwnode snd_pcm drm_kms_helper v4l2_async rapl
Sep 17 09:12:54 pve kernel:  intel_cstate pcspkr i2c_algo_bit cmdlinepart snd_timer mei_me videodev syscopyarea sysfillrect spi_nor snd mc mac_hid wmi_bmof acpi_pad acpi_tad mtd sysimgblt soundcore mei zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor uas usb_storage raid6_pq simplefb hid_generic usbhid hid dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme spi_intel_pci crc32_pclmul i2c_i801 nvme_core spi_intel i2c_smbus igc nvme_common sdhci_pci cqhci sdhci xhci_pci xhci_pci_renesas xhci_hcd video wmi pinctrl_alderlake
Sep 17 09:12:54 pve kernel: CPU: 1 PID: 0 Comm: swapper/1 Tainted: P           O       6.2.16-3-pve #1
Sep 17 09:12:54 pve kernel: Hardware name: CWWK CW-AD4L-N V1/CW-AD4L-N V1, BIOS 5.27 04/04/2023
Sep 17 09:12:54 pve kernel: RIP: 0010:dev_watchdog+0x23a/0x250
Sep 17 09:12:54 pve kernel: Code: 00 e9 2b ff ff ff 48 89 df c6 05 8a 6f 7d 01 01 e8 6b 08 f8 ff 44 89 f1 48 89 de 48 c7 c7 58 64 20 8c 48 89 c2 e8 06 ab 30 ff <0f> 0b e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00
Sep 17 09:12:54 pve kernel: RSP: 0018:ffffbd444007ce38 EFLAGS: 00010246
Sep 17 09:12:54 pve kernel: RAX: 0000000000000000 RBX: ffff9fd31281a000 RCX: 0000000000000000
Sep 17 09:12:54 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 17 09:12:54 pve kernel: RBP: ffffbd444007ce68 R08: 0000000000000000 R09: 0000000000000000
Sep 17 09:12:54 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff9fd31281a4c8
Sep 17 09:12:54 pve kernel: R13: ffff9fd31281a41c R14: 0000000000000000 R15: 0000000000000000
Sep 17 09:12:54 pve kernel: FS:  0000000000000000(0000) GS:ffff9fd66fa80000(0000) knlGS:0000000000000000
Sep 17 09:12:54 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 17 09:12:54 pve kernel: CR2: 000000c000161010 CR3: 0000000182610000 CR4: 0000000000752ee0
Sep 17 09:12:54 pve kernel: PKRU: 55555554
Sep 17 09:12:54 pve kernel: Call Trace:
Sep 17 09:12:54 pve kernel:  <IRQ>
Sep 17 09:12:54 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Sep 17 09:12:54 pve kernel:  call_timer_fn+0x29/0x160
Sep 17 09:12:54 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Sep 17 09:12:54 pve kernel:  __run_timers+0x259/0x310
Sep 17 09:12:54 pve kernel:  run_timer_softirq+0x1d/0x40
Sep 17 09:12:54 pve kernel:  __do_softirq+0xd6/0x346
Sep 17 09:12:54 pve kernel:  ? hrtimer_interrupt+0x11f/0x250
Sep 17 09:12:54 pve kernel:  __irq_exit_rcu+0xa2/0xd0
Sep 17 09:12:54 pve kernel:  irq_exit_rcu+0xe/0x20
Sep 17 09:12:54 pve kernel:  sysvec_apic_timer_interrupt+0x92/0xd0
Sep 17 09:12:54 pve kernel:  </IRQ>
Sep 17 09:12:54 pve kernel:  <TASK>
Sep 17 09:12:54 pve kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Sep 17 09:12:54 pve kernel: RIP: 0010:cpuidle_enter_state+0xde/0x6f0
Sep 17 09:12:54 pve kernel: Code: 2a b7 74 e8 54 7e 4a ff 8b 53 04 49 89 c7 0f 1f 44 00 00 31 ff e8 82 86 49 ff 80 7d d0 00 0f 85 eb 00 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 02 00 00 4d 63 ee 49 83 fd 09 0f 87 c7 04 00 00
Sep 17 09:12:54 pve kernel: RSP: 0018:ffffbd444016be38 EFLAGS: 00000246
Sep 17 09:12:54 pve kernel: RAX: 0000000000000000 RBX: ffffdd443fc80200 RCX: 0000000000000000
Sep 17 09:12:54 pve kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000
Sep 17 09:12:54 pve kernel: RBP: ffffbd444016be88 R08: 0000000000000000 R09: 0000000000000000
Sep 17 09:12:54 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff8ccc33a0
Sep 17 09:12:54 pve kernel: R13: 0000000000000004 R14: 0000000000000004 R15: 0000757e164668ed
Sep 17 09:12:54 pve kernel:  ? cpuidle_enter_state+0xce/0x6f0
Sep 17 09:12:54 pve kernel:  cpuidle_enter+0x2e/0x50
Sep 17 09:12:54 pve kernel:  do_idle+0x216/0x2a0
Sep 17 09:12:54 pve kernel:  cpu_startup_entry+0x1d/0x20
Sep 17 09:12:54 pve kernel:  start_secondary+0x122/0x160
Sep 17 09:12:54 pve kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Sep 17 09:12:54 pve kernel:  </TASK>
Sep 17 09:12:54 pve kernel: ---[ end trace 0000000000000000 ]---
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: Register Dump
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: Register Name   Value
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: CTRL            181c0641
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: STATUS          00280693
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: CTRL_EXT        100000c0
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: MDIC            180a3800
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: ICR             00000081
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: RCTL            0440803a
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: RDH[0-3]        000000fb 000000c0 0000008c 000000d2
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: RDT[0-3]        000000fa 000000bf 0000008b 000000d1
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: RDBAL[0-3]      050f5000 0586c000 06a51000 06a55000
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: TCTL            a503f0fa
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: TDBAL[0-3]      09b95000 07f77000 0a5b6000 1015f000
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: TDH[0-3]        00000004 0000002e 00000001 00000000
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: TDT[0-3]        00000010 00000037 00000001 00000000
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: Reset adapter
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 09:12:54 pve kernel: igc 0000:03:00.0 enp3s0: NIC Link is Down
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: Register Dump
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: Register Name   Value
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: CTRL            081c0641
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: STATUS          00280681
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: CTRL_EXT        100000c0
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: MDIC            18017949
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: ICR             00000001
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: RCTL            0440803a
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: RDH[0-3]        00000000 00000000 00000000 00000000
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: RDT[0-3]        000000ff 000000ff 000000ff 000000ff
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: RDBAL[0-3]      050f5000 0586c000 06a51000 06a55000
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: TCTL            a50400fa
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: TDBAL[0-3]      09b95000 07f77000 0a5b6000 1015f000
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: TDH[0-3]        00000008 00000000 00000000 00000000
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: TDT[0-3]        0000000d 000000eb 00000000 00000000
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 17 09:12:55 pve kernel: igc 0000:03:00.0 enp3s0: Reset adapter
Sep 17 09:12:55 pve kernel: vmbr1: port 1(enp3s0) entered disabled state
Sep 17 09:12:57 pve kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 09:12:57 pve kernel: vmbr1: port 1(enp3s0) entered blocking state
Sep 17 09:12:57 pve kernel: vmbr1: port 1(enp3s0) entered forwarding state
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: Register Dump
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: Register Name   Value
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: CTRL            181c0641
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: STATUS          00280693
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: CTRL_EXT        100000c0
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: MDIC            180a3800
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: ICR             00000081
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: RCTL            0440803a
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: RDH[0-3]        00000018 0000002e 0000006c 00000053
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: RDT[0-3]        00000017 0000002d 0000006b 00000052
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: RDBAL[0-3]      050f5000 0586c000 06a51000 06a55000
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: TCTL            a503f0fa
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: TDBAL[0-3]      09b95000 07f77000 0a5b6000 1015f000
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: TDH[0-3]        00000086 00000051 00000000 00000000
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: TDT[0-3]        0000008a 0000007a 00000000 00000000
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 17 09:13:23 pve kernel: igc 0000:03:00.0 enp3s0: Reset adapter
Sep 17 09:13:23 pve kernel: vmbr1: port 1(enp3s0) entered disabled state
Sep 17 09:13:26 pve kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 09:13:26 pve kernel: vmbr1: port 1(enp3s0) entered blocking state
Sep 17 09:13:26 pve kernel: vmbr1: port 1(enp3s0) entered forwarding state

The device was not accessible as far as I could tell via network so I had to power cycle to restore service.

Verbose pveversion

root@pve:~# pveversion --verbose
proxmox-ve: 8.0.1 (running kernel: 6.2.16-3-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.2
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.5
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.3
libpve-rs-perl: 0.8.3
libpve-storage-perl: 8.0.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 2.99.0-1
proxmox-backup-file-restore: 2.99.0-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.5
pve-cluster: 8.0.1
pve-container: 5.0.3
pve-docs: 8.0.3
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1

Update: Updated to 6.2.16-12-pve, will run for a bit and report back.

 

[weeeeems]

After a couple hours on the 6.2.16-12-pve it once again ceased to respond over ethernet

Sep 17 15:07:40 pve kernel: ------------[ cut here ]------------
Sep 17 15:07:40 pve kernel: NETDEV WATCHDOG: enp3s0 (igc): transmit queue 0 timed out
Sep 17 15:07:40 pve kernel: WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Sep 17 15:07:40 pve kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc nfnetlink_log nfnetlink binfmt_misc snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation snd_hda_codec_hdmi soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core intel_rapl_msr snd_soc_acpi_intel_match intel_rapl_common snd_soc_acpi soundwire_bus x86_pkg_temp_thermal intel_powerclamp coretemp snd_soc_core kvm_intel snd_compress ac97_bus i915 snd_pcm_dmaengine kvm drm_buddy snd_hda_intel ttm snd_intel_dspcfg snd_intel_sdw_acpi irqbypass drm_display_helper snd_hda_codec crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel cec sha512_ssse3 snd_hda_core rc_core aesni_intel ov13858 snd_hwdep crypto_simd v4l2_fwnode drm_kms_helper snd_pcm v4l2_async cryptd cmdlinepart snd_timer
Sep 17 15:07:40 pve kernel:  videodev i2c_algo_bit mei_me spi_nor snd syscopyarea sysfillrect rapl intel_cstate pcspkr wmi_bmof mtd soundcore sysimgblt mei mc acpi_tad acpi_pad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor uas usb_storage raid6_pq hid_generic usbhid hid simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme sdhci_pci xhci_pci nvme_core xhci_pci_renesas cqhci i2c_i801 spi_intel_pci sdhci i2c_smbus nvme_common crc32_pclmul spi_intel video xhci_hcd igc wmi pinctrl_alderlake
Sep 17 15:07:40 pve kernel: CPU: 1 PID: 0 Comm: swapper/1 Tainted: P           O       6.2.16-12-pve #1
Sep 17 15:07:40 pve kernel: Hardware name: CWWK CW-AD4L-N V1/CW-AD4L-N V1, BIOS 5.27 04/04/2023
Sep 17 15:07:40 pve kernel: RIP: 0010:dev_watchdog+0x23a/0x250
Sep 17 15:07:40 pve kernel: Code: 00 e9 2b ff ff ff 48 89 df c6 05 cc 66 7d 01 01 e8 db 08 f8 ff 44 89 f1 48 89 de 48 c7 c7 b8 78 c0 86 48 89 c2 e8 06 9c 30 ff <0f> 0b e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00
Sep 17 15:07:40 pve kernel: RSP: 0018:ffff9905c007ce38 EFLAGS: 00010246
Sep 17 15:07:40 pve kernel: RAX: 0000000000000000 RBX: ffff8ae012132000 RCX: 0000000000000000
Sep 17 15:07:40 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 17 15:07:40 pve kernel: RBP: ffff9905c007ce68 R08: 0000000000000000 R09: 0000000000000000
Sep 17 15:07:40 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8ae0121324c8
Sep 17 15:07:40 pve kernel: R13: ffff8ae01213241c R14: 0000000000000000 R15: 0000000000000000
Sep 17 15:07:40 pve kernel: FS:  0000000000000000(0000) GS:ffff8ae36fa80000(0000) knlGS:0000000000000000
Sep 17 15:07:40 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 17 15:07:40 pve kernel: CR2: 00001ea51c33fd90 CR3: 000000021b010000 CR4: 0000000000752ee0
Sep 17 15:07:40 pve kernel: PKRU: 55555554
Sep 17 15:07:40 pve kernel: Call Trace:
Sep 17 15:07:40 pve kernel:  <IRQ>
Sep 17 15:07:40 pve kernel:  ? show_regs+0x6d/0x80
Sep 17 15:07:40 pve kernel:  ? __warn+0x89/0x160
Sep 17 15:07:40 pve kernel:  ? dev_watchdog+0x23a/0x250
Sep 17 15:07:40 pve kernel:  ? report_bug+0x17e/0x1b0
Sep 17 15:07:40 pve kernel:  ? irq_work_queue+0x2f/0x70
Sep 17 15:07:40 pve kernel:  ? handle_bug+0x46/0x90
Sep 17 15:07:40 pve kernel:  ? exc_invalid_op+0x18/0x80
Sep 17 15:07:40 pve kernel:  ? asm_exc_invalid_op+0x1b/0x20
Sep 17 15:07:40 pve kernel:  ? dev_watchdog+0x23a/0x250
Sep 17 15:07:40 pve kernel:  ? dev_watchdog+0x23a/0x250
Sep 17 15:07:40 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Sep 17 15:07:40 pve kernel:  call_timer_fn+0x29/0x160
Sep 17 15:07:40 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Sep 17 15:07:40 pve kernel:  __run_timers+0x259/0x310
Sep 17 15:07:40 pve kernel:  run_timer_softirq+0x1d/0x40
Sep 17 15:07:40 pve kernel:  __do_softirq+0xd6/0x346
Sep 17 15:07:40 pve kernel:  ? hrtimer_interrupt+0x11f/0x250
Sep 17 15:07:40 pve kernel:  __irq_exit_rcu+0xa2/0xd0
Sep 17 15:07:40 pve kernel:  irq_exit_rcu+0xe/0x20
Sep 17 15:07:40 pve kernel:  sysvec_apic_timer_interrupt+0x92/0xd0
Sep 17 15:07:40 pve kernel:  </IRQ>
Sep 17 15:07:40 pve kernel:  <TASK>
Sep 17 15:07:40 pve kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Sep 17 15:07:40 pve kernel: RIP: 0010:cpuidle_enter_state+0xde/0x6f0
Sep 17 15:07:40 pve kernel: Code: 1c 17 7a e8 54 6f 4a ff 8b 53 04 49 89 c7 0f 1f 44 00 00 31 ff e8 82 77 49 ff 80 7d d0 00 0f 85 eb 00 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 02 00 00 4d 63 ee 49 83 fd 09 0f 87 c7 04 00 00
Sep 17 15:07:40 pve kernel: RSP: 0018:ffff9905c016be38 EFLAGS: 00000246
Sep 17 15:07:40 pve kernel: RAX: 0000000000000000 RBX: ffffb905bfc80200 RCX: 0000000000000000
Sep 17 15:07:40 pve kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000
Sep 17 15:07:40 pve kernel: RBP: ffff9905c016be88 R08: 0000000000000000 R09: 0000000000000000
Sep 17 15:07:40 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff876c39a0
Sep 17 15:07:40 pve kernel: R13: 0000000000000004 R14: 0000000000000004 R15: 0000045741350f32
Sep 17 15:07:40 pve kernel:  ? cpuidle_enter_state+0xce/0x6f0
Sep 17 15:07:40 pve kernel:  cpuidle_enter+0x2e/0x50
Sep 17 15:07:40 pve kernel:  do_idle+0x216/0x2a0
Sep 17 15:07:40 pve kernel:  cpu_startup_entry+0x1d/0x20
Sep 17 15:07:40 pve kernel:  start_secondary+0x122/0x160
Sep 17 15:07:40 pve kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Sep 17 15:07:40 pve kernel:  </TASK>
Sep 17 15:07:40 pve kernel: ---[ end trace 0000000000000000 ]---
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: Register Dump
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: Register Name   Value
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: CTRL            181c0641
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: STATUS          00280693
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: CTRL_EXT        100000c0
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: MDIC            180a3800
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: ICR             00000081
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: RCTL            0440803a
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: RDH[0-3]        0000006f 00000065 00000071 00000072
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: RDT[0-3]        0000006e 00000064 00000070 00000071
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: RDBAL[0-3]      06e11000 1b854000 06ae1000 07cac000
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: TCTL            a503f0fa
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: TDBAL[0-3]      0dbfb000 0ebab000 06c4a000 06e3c000
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: TDH[0-3]        0000006a 00000083 00000000 00000002
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: TDT[0-3]        00000096 00000086 00000000 00000002
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 17 15:07:40 pve kernel: igc 0000:03:00.0 enp3s0: Reset adapter
Sep 17 15:07:40 pve kernel: vmbr1: port 1(enp3s0) entered disabled state
Sep 17 15:07:43 pve kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 15:07:43 pve kernel: vmbr1: port 1(enp3s0) entered blocking state
Sep 17 15:07:43 pve kernel: vmbr1: port 1(enp3s0) entered forwarding state
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: Register Dump
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: Register Name   Value
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: CTRL            181c0641
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: STATUS          00280693
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: CTRL_EXT        100000c0
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: MDIC            180a3800
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: ICR             00000081
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: RCTL            0440803a
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: RDH[0-3]        00000056 0000000c 0000001b 00000000
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: RDT[0-3]        00000055 0000000b 0000001a 000000ff
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: RDBAL[0-3]      06e11000 1b854000 06ae1000 07cac000
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: TCTL            a503f0fa
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: TDBAL[0-3]      0dbfb000 0ebab000 06c4a000 06e3c000
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: TDH[0-3]        00000051 00000008 00000000 00000000
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: TDT[0-3]        00000061 0000000a 00000000 00000000
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 17 15:07:59 pve kernel: igc 0000:03:00.0 enp3s0: Reset adapter
Sep 17 15:07:59 pve kernel: vmbr1: port 1(enp3s0) entered disabled state
Sep 17 15:08:06 pve kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 15:08:06 pve kernel: vmbr1: port 1(enp3s0) entered blocking state
Sep 17 15:08:06 pve kernel: vmbr1: port 1(enp3s0) entered forwarding state

Updated versions from shell

root@pve:~# lspci -nn | grep Ethernet
02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
root@pve:~# pveversion
pve-manager/8.0.6/57490ff2c6a38448 (running kernel: 6.2.16-12-pve)
root@pve:~# pveversion --verbose
proxmox-ve: 8.0.2 (running kernel: 6.2.16-12-pve)
pve-manager: 8.0.6 (running version: 8.0.6/57490ff2c6a38448)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-12-pve: 6.2.16-12
proxmox-kernel-6.2: 6.2.16-12
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.9
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.6
libpve-storage-perl: 8.0.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.8
pve-cluster: 8.0.4
pve-container: 5.0.4
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1