[SOLVED] Another kernel BUG at lib/dynamic_queue_limits.c:27!

[tictag]

I have been experiencing random crashes of Proxmox v8.0.4 (Linux version 6.2.16-3-pve) for many months since installing it on a Tuofudun 2.5GbE Firewall Mirco Appliance. It has taken a significant effort but I now believe I have at least one repeatable cause ... every time my Windows laptop resumes from sleep, Proxmox crashes. Please find an example of the kernel syslog below:

.........
Aug 13 17:05:03 pve pvestatd[924]: auth key pair too old, rotating..
Aug 13 17:17:01 pve CRON[196694]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Aug 13 17:17:01 pve CRON[196695]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Aug 13 17:17:01 pve CRON[196694]: pam_unix(cron:session): session closed for user root
Aug 13 18:17:01 pve CRON[206380]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Aug 13 18:17:01 pve CRON[206381]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Aug 13 18:17:01 pve CRON[206380]: pam_unix(cron:session): session closed for user root
Aug 13 18:34:23 pve kernel: perf: interrupt took too long (3919 > 3917), lowering kernel.perf_event_max_sample_rate to 51000

WINDOWS LAPTOP RESUMES FROM SLEEP

Aug 13 18:59:48 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Aug 13 18:59:48 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Aug 13 18:59:48 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Aug 13 18:59:52 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Aug 13 18:59:52 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Aug 13 18:59:52 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Aug 13 19:00:03 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Aug 13 19:00:03 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Aug 13 19:00:03 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Aug 13 19:00:04 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Aug 13 19:00:07 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Aug 13 19:00:07 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Aug 13 19:00:07 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Aug 13 19:00:07 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Aug 13 19:00:07 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Aug 13 19:00:07 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Aug 13 19:02:44 pve kernel: ------------[ cut here ]------------
Aug 13 19:02:44 pve kernel: refcount_t: underflow; use-after-free.
Aug 13 19:02:44 pve kernel: WARNING: CPU: 3 PID: 0 at lib/refcount.c:28 refcount_warn_saturate+0xa3/0x150
Aug 13 19:02:44 pve kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc nfnetlink_log binfmt_misc nfnetlink snd_hda_codec_hdmi snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus x86_pkg_temp_thermal intel_powerclamp i915 coretemp snd_soc_core snd_compress drm_buddy ac97_bus snd_pcm_dmaengine ttm kvm_intel drm_display_helper cec snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi kvm cmdlinepart crct10dif_pclmul snd_hda_codec polyval_generic ghash_clmulni_intel sha512_ssse3 rc_core snd_hda_core spi_nor aesni_intel intel_rapl_msr drm_kms_helper crypto_simd snd_hwdep processor_thermal_device_pci_legacy i2c_algo_bit cryptd processor_thermal_device snd_pcm
Aug 13 19:02:44 pve kernel:  processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common mei_me snd_timer int340x_thermal_zone syscopyarea snd sysfillrect intel_cstate mtd pcspkr wmi_bmof sysimgblt soundcore mei intel_soc_dts_iosf zfs(PO) mac_hid zunicode(PO) acpi_pad acpi_tad zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb uas usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c i2c_i801 spi_intel_pci crc32_pclmul spi_intel i2c_smbus igc sdhci_pci nvme xhci_pci cqhci xhci_pci_renesas sdhci nvme_core nvme_common xhci_hcd video wmi pinctrl_jasperlake
Aug 13 19:02:44 pve kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P        W  O       6.2.16-3-pve #1
Aug 13 19:02:44 pve kernel: Hardware name: Default string Default string/Default string, BIOS 5.19 11/11/2022
Aug 13 19:02:44 pve kernel: RIP: 0010:refcount_warn_saturate+0xa3/0x150
Aug 13 19:02:44 pve kernel: Code: cc cc 0f b6 1d 30 7f e0 01 80 fb 01 0f 87 e9 8b 88 00 83 e3 01 75 dd 48 c7 c7 08 e5 96 9d c6 05 14 7f e0 01 01 e8 ad bb 93 ff <0f> 0b eb c6 0f b6 1d 07 7f e0 01 80 fb 01 0f 87 a9 8b 88 00 83 e3
Aug 13 19:02:44 pve kernel: RSP: 0018:ffffb49b401d4d78 EFLAGS: 00010246
Aug 13 19:02:44 pve kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Aug 13 19:02:44 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Aug 13 19:02:44 pve kernel: RBP: ffffb49b401d4d80 R08: 0000000000000000 R09: 0000000000000000
Aug 13 19:02:44 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff913bd18a5c00
Aug 13 19:02:44 pve kernel: R13: ffff913bca2afff0 R14: ffff913bca2af000 R15: 00000000ffffffff
Aug 13 19:02:44 pve kernel: FS:  0000000000000000(0000) GS:ffff913d38180000(0000) knlGS:0000000000000000
Aug 13 19:02:44 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 13 19:02:44 pve kernel: CR2: 00003d7f66d00a20 CR3: 0000000116eca000 CR4: 0000000000352ee0
Aug 13 19:02:44 pve kernel: Call Trace:
Aug 13 19:02:44 pve kernel:  <IRQ>
Aug 13 19:02:44 pve kernel:  napi_consume_skb+0x15b/0x180
Aug 13 19:02:44 pve kernel:  igc_poll+0x874/0x17d0 [igc]
Aug 13 19:02:44 pve kernel:  ? __mod_timer+0x28c/0x400
Aug 13 19:02:44 pve kernel:  __napi_poll+0x30/0x1f0
Aug 13 19:02:44 pve kernel:  net_rx_action+0x180/0x2d0
Aug 13 19:02:44 pve kernel:  ? __napi_schedule+0x71/0xa0
Aug 13 19:02:44 pve kernel:  __do_softirq+0xd6/0x346
Aug 13 19:02:44 pve kernel:  ? handle_irq_event+0x52/0x80
Aug 13 19:02:44 pve kernel:  ? handle_edge_irq+0xda/0x250
Aug 13 19:02:44 pve kernel:  __irq_exit_rcu+0xa2/0xd0
Aug 13 19:02:44 pve kernel:  irq_exit_rcu+0xe/0x20
Aug 13 19:02:44 pve kernel:  common_interrupt+0xa4/0xb0
Aug 13 19:02:44 pve kernel:  </IRQ>
Aug 13 19:02:44 pve kernel:  <TASK>
Aug 13 19:02:44 pve kernel:  asm_common_interrupt+0x27/0x40
Aug 13 19:02:44 pve kernel: RIP: 0010:native_safe_halt+0xb/0x10
Aug 13 19:02:44 pve kernel: Code: 20 5f 25 9e e8 a6 ee 7d ff e9 3e ff ff ff cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 90 0f 00 2d a9 d4 37 00 fb f4 <c3> cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66
Aug 13 19:02:44 pve kernel: RSP: 0018:ffffb49b4012fde0 EFLAGS: 00000246
Aug 13 19:02:44 pve kernel: RAX: 0000000000004800 RBX: ffff913bc12cf464 RCX: 0000000000000000
Aug 13 19:02:44 pve kernel: RDX: 0000000000000001 RSI: ffff913bc12cf400 RDI: 0000000000000001
Aug 13 19:02:44 pve kernel: RBP: ffffb49b4012fdf0 R08: 0000000000000000 R09: 0000000000000000
Aug 13 19:02:44 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff913bc12cf464
Aug 13 19:02:44 pve kernel: R13: 0000000000000003 R14: ffffffff9e4d6600 R15: ffff913d38180000
Aug 13 19:02:44 pve kernel:  ? acpi_idle_do_entry+0x82/0xc0
Aug 13 19:02:44 pve kernel:  acpi_idle_enter+0xbb/0x180
Aug 13 19:02:44 pve kernel:  cpuidle_enter_state+0x9a/0x6f0
Aug 13 19:02:44 pve kernel:  cpuidle_enter+0x2e/0x50
Aug 13 19:02:44 pve kernel:  do_idle+0x216/0x2a0
Aug 13 19:02:44 pve kernel:  cpu_startup_entry+0x1d/0x20
Aug 13 19:02:44 pve kernel:  start_secondary+0x122/0x160
Aug 13 19:02:44 pve kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Aug 13 19:02:44 pve kernel:  </TASK>
Aug 13 19:02:44 pve kernel: ---[ end trace 0000000000000000 ]---
Aug 13 19:02:44 pve kernel: ------------[ cut here ]------------
Aug 13 19:02:44 pve kernel: kernel BUG at lib/dynamic_queue_limits.c:27!
-- Reboot --
Aug 13 19:06:33 pve kernel: Linux version 6.2.16-3-pve (tom@sbuild) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) ()
Aug 13 19:06:33 pve kernel: Command line: B
.........

Please also find the output from lspci -nn | grep Ethernet below:

02:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)

I suspect that there is either a hardware problem with the Intel I226-V NIC or a software problem with its kernel driver.

I am new to Proxmox and, in fact, Linux so I don't know how much help I can be but if you could suggest a troubleshooting strategy then I will do my best to follow.

Regards,
David.
edit: correct typo.

 

[fiona]

Hi,
sounds like the same issue reported here: https://forum.proxmox.com/threads/kernel-bug-at-lib-dynamic_queue_limits-c-27.130935/

 

[tictag]

Yeah, that's the one that led me to put 'another' in the post title; different cause, different NIC but same error. The previous poster's troubleshooting strategy was to wait until a patch was released, I was wondering with this post whether an alternative strategy existed.

 

[tictag]

Following on from post #12, updated Proxmox Linux Kernel to V6.2.16-9.

It was my belief that the kernel was crashing every time my computer woke up from sleep, but this hypothesis is flawed:

• Computer woke up from sleep on its own - kernel crashes
• Computer wakes up from sleep because I wake it - kernel does not crash

Mind blown

****    UPDATE KERNEL TO V6.2.16-9    ****
...
...
...
Sep 01 13:37:52 pve pvedaemon[1279]: command 'apt-get update' failed: exit code 100
Sep 01 13:37:52 pve pvedaemon[953]: <root@pam> end task UPID:pve:000004FF:00001704:64F1DB1F:aptupdate::root@pam: command 'apt-get update' failed: exit code 100
Sep 01 13:38:12 pve pvedaemon[952]: update new package list: /var/lib/pve-manager/pkgupdates


ZZZZ   DAVID PUTS COMPUTER TO SLEEP AND TAKES A NAP HIMSELF    ZZZZ


Sep 01 13:41:17 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Sep 01 13:41:17 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Sep 01 13:41:17 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Sep 01 13:41:17 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Sep 01 13:52:32 pve systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Sep 01 13:52:32 pve systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Sep 01 13:52:32 pve systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Sep 01 13:52:32 pve systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Sep 01 14:06:56 pve smartd[594]: Device: /dev/sda [SAT], removed ATA device: No such device


****    COMPUTER WAKES UP ON ITS OWN    ****


Sep 01 14:12:10 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 01 14:12:10 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 01 14:12:10 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 01 14:12:10 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 01 14:12:10 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 01 14:12:10 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state


****    KERNEL CRASHES    ****


Sep 01 14:12:31 pve kernel: ------------[ cut here ]------------
Sep 01 14:12:31 pve kernel: NETDEV WATCHDOG: enp5s0 (igc): transmit queue 0 timed out
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: Register Dump
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: Register Name   Value
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: CTRL            181c0641
Sep 01 14:12:31 pve kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: STATUS          00680693
Sep 01 14:12:31 pve kernel: Modules linked in: ebtable_filter ebtables
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: CTRL_EXT        10000040
Sep 01 14:12:31 pve kernel:  ip_set ip6table_raw iptable_raw ip6table_filter
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: MDIC            1805dde1
Sep 01 14:12:31 pve kernel:  ip6_tables iptable_filter bpfilter
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: ICR             00000081
Sep 01 14:12:31 pve kernel:  nf_tables sunrpc bonding tls
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: RCTL            0440803a
Sep 01 14:12:31 pve kernel:  softdog nfnetlink_log nfnetlink binfmt_misc snd_hda_codec_hdmi snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 01 14:12:31 pve kernel:  snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: RDH[0-3]        0000009f 0000004a 00000053 000000d2
Sep 01 14:12:31 pve kernel:  i915 x86_pkg_temp_thermal snd_soc_core zfs(PO) intel_powerclamp drm_buddy snd_compress ttm ac97_bus snd_pcm_dmaengine
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: RDT[0-3]        0000009e 00000049 00000052 000000d1
Sep 01 14:12:31 pve kernel:  coretemp drm_display_helper snd_hda_intel cec kvm_intel zunicode(PO) snd_intel_dspcfg snd_intel_sdw_acpi zzstd(O) processor_thermal_device_pci_legacy
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 01 14:12:31 pve kernel:  rc_core zlua(O) zavl(PO) icp(PO) snd_hda_codec processor_thermal_device kvm processor_thermal_rfim intel_rapl_msr zcommon(PO)
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: RDBAL[0-3]      0627e000 052c1000 052c5000 052c8000
Sep 01 14:12:31 pve kernel:  processor_thermal_mbox crct10dif_pclmul znvpair(PO) snd_hda_core polyval_generic ghash_clmulni_intel snd_hwdep sha512_ssse3 spl(O)
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 01 14:12:31 pve kernel:  aesni_intel snd_pcm vhost_net vhost
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: TCTL            a503f0fa
Sep 01 14:12:31 pve kernel:  crypto_simd snd_timer processor_thermal_rapl vhost_iotlb tap cmdlinepart snd cryptd intel_rapl_common vfio_pci drm_kms_helper
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: TDBAL[0-3]      0626b000 06270000 06276000 0627b000
Sep 01 14:12:31 pve kernel:  vfio_pci_core irqbypass spi_nor vfio_iommu_type1 i2c_algo_bit syscopyarea mei_me sysfillrect int340x_thermal_zone vfio intel_cstate
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 01 14:12:31 pve kernel:  wmi_bmof pcspkr mei soundcore sysimgblt mtd intel_soc_dts_iosf acpi_pad acpi_tad mac_hid iommufd drm
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 01 14:12:31 pve kernel:  efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: TDH[0-3]        00000001 000000ba 0000006f 000000b5
Sep 01 14:12:31 pve kernel:  uas usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci nvme xhci_pci_renesas sdhci_pci
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: TDT[0-3]        0000000d 000000be 0000007b 000000ba
Sep 01 14:12:31 pve kernel:  xhci_hcd crc32_pclmul i2c_i801 spi_intel_pci igc spi_intel nvme_core i2c_smbus cqhci nvme_common sdhci
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 01 14:12:31 pve kernel:  video wmi
Sep 01 14:12:31 pve kernel: igc 0000:04:00.0 enp4s0: Reset adapter
Sep 01 14:12:31 pve kernel:  pinctrl_jasperlake
Sep 01 14:12:31 pve kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           O       6.2.16-10-pve #1
Sep 01 14:12:31 pve kernel: Hardware name: Default string Default string/Default string, BIOS 5.19 11/11/2022
Sep 01 14:12:31 pve kernel: RIP: 0010:dev_watchdog+0x23a/0x250
Sep 01 14:12:31 pve kernel: Code: 00 e9 2b ff ff ff 48 89 df c6 05 8c 68 7d 01 01 e8 6b 08 f8 ff 44 89 f1 48 89 de 48 c7 c7 b0 6e 20 8d 48 89 c2 e8 86 9f 30 ff <0f> 0b e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00
Sep 01 14:12:31 pve kernel: RSP: 0018:ffff9d9740003e38 EFLAGS: 00010246
Sep 01 14:12:31 pve kernel: RAX: 0000000000000000 RBX: ffff904091716000 RCX: 0000000000000000
Sep 01 14:12:31 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 01 14:12:31 pve kernel: RBP: ffff9d9740003e68 R08: 0000000000000000 R09: 0000000000000000
Sep 01 14:12:31 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff9040917164c8
Sep 01 14:12:31 pve kernel: R13: ffff90409171641c R14: 0000000000000000 R15: 0000000000000000
Sep 01 14:12:31 pve kernel: FS:  0000000000000000(0000) GS:ffff9041f8000000(0000) knlGS:0000000000000000
Sep 01 14:12:31 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 01 14:12:31 pve kernel: CR2: 000000c000865000 CR3: 00000001059e0000 CR4: 0000000000352ef0
Sep 01 14:12:31 pve kernel: Call Trace:
Sep 01 14:12:31 pve kernel:  <IRQ>
Sep 01 14:12:31 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Sep 01 14:12:31 pve kernel:  call_timer_fn+0x29/0x160
Sep 01 14:12:31 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Sep 01 14:12:31 pve kernel:  __run_timers+0x259/0x310
Sep 01 14:12:31 pve kernel:  run_timer_softirq+0x1d/0x40
Sep 01 14:12:31 pve kernel:  __do_softirq+0xd6/0x346
Sep 01 14:12:31 pve kernel:  ? hrtimer_interrupt+0x11f/0x250
Sep 01 14:12:31 pve kernel:  __irq_exit_rcu+0xa2/0xd0
Sep 01 14:12:31 pve kernel:  irq_exit_rcu+0xe/0x20
Sep 01 14:12:31 pve kernel:  sysvec_apic_timer_interrupt+0x92/0xd0
Sep 01 14:12:31 pve kernel:  </IRQ>
Sep 01 14:12:31 pve kernel:  <TASK>
Sep 01 14:12:31 pve kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Sep 01 14:12:31 pve kernel: RIP: 0010:native_safe_halt+0xb/0x10
Sep 01 14:12:31 pve kernel: Code: e0 5f a5 8d e8 a6 e8 7d ff e9 3e ff ff ff cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 90 0f 00 2d 09 c5 37 00 fb f4 <c3> cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66
Sep 01 14:12:31 pve kernel: RSP: 0018:ffffffff8da03d50 EFLAGS: 00000246
Sep 01 14:12:31 pve kernel: RAX: 0000000000004800 RBX: ffff9040812c9c64 RCX: 0000000000000000
Sep 01 14:12:31 pve kernel: RDX: 0000000000000001 RSI: ffff9040812c9c00 RDI: 0000000000000001
Sep 01 14:12:31 pve kernel: RBP: ffffffff8da03d60 R08: 0000000000000000 R09: 0000000000000000
Sep 01 14:12:31 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff9040812c9c64
Sep 01 14:12:31 pve kernel: R13: 0000000000000000 R14: ffffffff8dcd6920 R15: ffff9041f8000000
Sep 01 14:12:31 pve kernel:  ? acpi_idle_do_entry+0x82/0xc0
Sep 01 14:12:31 pve kernel:  acpi_idle_enter+0xbb/0x180
Sep 01 14:12:31 pve kernel:  cpuidle_enter_state+0x9a/0x6f0
Sep 01 14:12:31 pve kernel:  cpuidle_enter+0x2e/0x50
Sep 01 14:12:31 pve kernel:  do_idle+0x216/0x2a0
Sep 01 14:12:31 pve kernel:  cpu_startup_entry+0x1d/0x20
Sep 01 14:12:31 pve kernel:  rest_init+0xdc/0x100
Sep 01 14:12:31 pve kernel:  ? acpi_enable_subsystem+0xe6/0x2a0
Sep 01 14:12:31 pve kernel:  ? static_key_disable+0x1f/0x30
Sep 01 14:12:31 pve kernel:  arch_call_rest_init+0xe/0x30
Sep 01 14:12:31 pve kernel:  start_kernel+0x6ab/0xb80
Sep 01 14:12:31 pve kernel:  ? load_ucode_intel_bsp+0x3d/0x80
Sep 01 14:12:31 pve kernel:  x86_64_start_kernel+0x102/0x180
Sep 01 14:12:31 pve kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Sep 01 14:12:31 pve kernel:  </TASK>
Sep 01 14:12:31 pve kernel: ---[ end trace 0000000000000000 ]---
Sep 01 14:12:31 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: Register Dump
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: Register Name   Value
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: CTRL            181c0641
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: STATUS          00680693
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: CTRL_EXT        10000040
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: MDIC            1805dde1
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: ICR             00000081
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: RCTL            0440803a
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: RDH[0-3]        00000099 000000df 0000000b 0000006e
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: RDT[0-3]        00000098 000000de 0000000a 0000006d
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: RDBAL[0-3]      06ffd000 06f40000 06f43000 06f46000
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: TCTL            a503f0fa
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: TDBAL[0-3]      06feb000 06ff0000 06ff5000 06ffa000
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: TDH[0-3]        0000004e 000000a1 000000e9 0000002a
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: TDT[0-3]        00000039 000000e3 000000fd 0000002f
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 01 14:12:31 pve kernel: igc 0000:05:00.0 enp5s0: Reset adapter
Sep 01 14:12:32 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Sep 01 14:12:35 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 01 14:12:35 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 01 14:12:35 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 01 14:12:35 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 01 14:12:35 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 01 14:12:35 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state


****   FLATMATE POWER RECYCLES NUC    ****


-- Reboot --
Sep 01 14:18:20 pve kernel: Linux version 6.2.16-10-pve (wolfgangb@sbuild) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-10 (2023-08-18T11:42Z) ()
Sep 01 14:18:20 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-10-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
Sep 01 14:18:20 pve kernel: KERNEL supported cpus:
...
...
...
Sep 01 16:17:01 pve CRON[20063]: pam_unix(cron:session): session closed for user root
Sep 01 17:06:20 pve pvestatd[925]: auth key pair too old, rotating..
Sep 01 17:17:01 pve CRON[29605]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 01 17:17:01 pve CRON[29606]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 01 17:17:01 pve CRON[29605]: pam_unix(cron:session): session closed for user root


ZZZZ    DAVID WAKES UP FROM HIS NAP AND TURNS COMPUTER ON    ZZZZ


Sep 01 18:09:59 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 01 18:09:59 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 01 18:09:59 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 01 18:10:02 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 01 18:10:02 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 01 18:10:02 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Sep 01 18:10:15 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Sep 01 18:10:15 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Sep 01 18:10:15 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Sep 01 18:10:16 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Sep 01 18:10:18 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 01 18:10:18 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 01 18:10:18 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 01 18:10:18 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 01 18:10:18 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 01 18:10:18 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state


****    NO KERNEL CRASH    ****


Sep 01 18:11:47 pve pvedaemon[954]: <root@pam> successful auth for user 'root@pam'
Sep 01 18:17:01 pve CRON[39426]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 01 18:17:01 pve CRON[39427]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 01 18:17:01 pve CRON[39426]: pam_unix(cron:session): session closed for user root
Sep 01 18:26:40 pve pvedaemon[956]: <root@pam> successful auth for user 'root@pam'

 

[tictag]

It seems obvious to me that the NIC (Intel I226-V) or its driver are causing this.

Is there a way I can report this to Intel?

 

[fweber]

Thank you for sharing your observations!

It was my belief that the kernel was crashing every time my computer woke up from sleep, but this hypothesis is flawed:

• Computer woke up from sleep on its own - kernel crashes
• Computer wakes up from sleep because I wake it - kernel does not crash

Mind blown

This is indeed weird. Can you reproduce this reliably / do the crashes still occur? Just out of interest, could you share your /etc/network/interfaces?

Is there a way I can report this to Intel?

Probably -- but personally I don't know how to best reach Intel regarding these kinds of issues, sorry.

 

[tictag]

This is indeed weird. Can you reproduce this reliably / do the crashes still occur?

Since updating to Linux version 6.2.16-3-pve I have been unable to recreate the issue where if I manually put my computer to sleep, then wake it up it always crashes the kernel, which is good news! I don't know under what conditions the computer wakes itself up, so I will test over a few nights and check the logs when I get up.

Just out of interest, could you share your /etc/network/interfaces

As requested:

root@pve:~# nano /etc/network/interfaces
  GNU nano 7.2                                                       /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp3s0 inet manual

iface enp4s0 inet manual

iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.3/24
        gateway 192.168.1.1
        bridge-ports enp3s0 enp4s0 enp5s0
        bridge-stp off
        bridge-fd 0

My configuration is pretty basic: Proxmox running on 4-port NUC (Intel I226-V NICs), enp2s0 connects to my ISP and is passed through to pfSense's WAN interface (100.x.x.x/26), the remaining three are bridged to pfSense's LAN interface (192.168.1.0/24). Nothing fancy, no VLANs, no network segregation etc.

 

[fiona]

every time my Windows laptop resumes from sleep, Proxmox crashes.

What exactly is the relation between the Windows laptop and Proxmox here? Are they just running as two separate physical machines on the same network or is it something else?

 

[tictag]

What exactly is the relation between the Windows laptop and Proxmox here?

Thank you for the question. My windows laptop is simply connected to the LAN created and managed by my NUC device, I use it to, for example, browse the management interfaces for proxmox and my virtual machines. I have a few other devices, like my smartphone, smartTV etc connected via a WiFi AP but my laptop is hard-wired (Ethernet CAT6) to bridged ports enp4s0 and enp5s0 in order to take advantage of SMB3 MultiChannel connection to my NAS.

Put simply, my laptop is just another device connected to my network.


Edit: please forgive the horrible handwriting!!

 

[tictag]

I don't know under what conditions the computer wakes itself up, so I will test over a few nights and check the logs when I get up.

Put computer to sleep last night, it did not wake itself up, resumed without a crash this morning. Test continues...

 

[tictag]

Now multiple days where computer was put to sleep but did not automatically wake up until the power button was pressed again the following morning. Multiple sleep/wake events now and NO further crashes. Looks like Linux version 6.2.16-3-pve has resolved that issue.

 

[tictag]

Finally, last night (or rather this morning) my computer automatically woke up - and the kernel did NOT crash, though it did record 'something different' to when the computer is manually woken up. See system log below.

Sep 11 04:36:04 pve systemd[1]: pve-daily-update.service: Deactivated successfully.
Sep 11 04:36:04 pve systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
Sep 11 04:36:04 pve systemd[1]: pve-daily-update.service: Consumed 4.575s CPU time.


*** COMPUTER MANUALLY PUT TO SLEEP ***


Sep 11 05:07:17 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Sep 11 05:07:17 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Sep 11 05:07:23 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Sep 11 05:07:23 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Sep 11 05:17:01 pve CRON[788357]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 05:17:01 pve CRON[788358]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 05:17:01 pve CRON[788357]: pam_unix(cron:session): session closed for user root


*** COMPUTER AUTOMATICALLY WOKE UP ***


Sep 11 05:38:09 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 11 05:38:09 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 11 05:38:09 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 11 05:38:09 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 11 05:38:09 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 11 05:38:09 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Sep 11 05:38:25 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Sep 11 05:38:25 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Sep 11 05:38:25 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: Register Dump
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: Register Name   Value
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: CTRL            181c0641
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: STATUS          00680681
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: CTRL_EXT        10000040
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: MDIC            18017949
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: ICR             00000000
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: RCTL            0440803a
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: RDH[0-3]        000000a3 00000023 0000006f 00000054
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: RDT[0-3]        000000a2 00000022 0000006e 00000053
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: RDBAL[0-3]      06ac8000 06acb000 06ace000 06ad1000
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: TCTL            a503f0fa
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: TDBAL[0-3]      06ab6000 06abb000 06ac0000 06ac5000
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: TDH[0-3]        000000ad 0000008e 000000c7 000000ec
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: TDT[0-3]        000000eb 00000092 000000cc 000000ec
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 11 05:38:25 pve kernel: igc 0000:05:00.0 enp5s0: Reset adapter


*** COMPUTER AUTOMATICALLY WENT TO SLEEP ***


Sep 11 06:17:01 pve CRON[797646]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 06:17:01 pve CRON[797647]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 06:17:01 pve CRON[797646]: pam_unix(cron:session): session closed for user root
Sep 11 06:25:01 pve CRON[798909]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 06:25:01 pve CRON[798910]: (root) CMD (test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; })
Sep 11 06:25:01 pve CRON[798909]: pam_unix(cron:session): session closed for user root
Sep 11 06:30:55 pve systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
Sep 11 06:30:56 pve systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
Sep 11 06:30:56 pve systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
Sep 11 07:17:01 pve CRON[807009]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 07:17:01 pve CRON[807010]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 07:17:01 pve CRON[807009]: pam_unix(cron:session): session closed for user root
Sep 11 08:17:01 pve CRON[816365]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 08:17:01 pve CRON[816366]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 08:17:01 pve CRON[816365]: pam_unix(cron:session): session closed for user root
Sep 11 09:14:55 pve systemd[1]: Starting apt-daily.service - Daily apt download activities...
Sep 11 09:14:55 pve systemd[1]: apt-daily.service: Deactivated successfully.
Sep 11 09:14:55 pve systemd[1]: Finished apt-daily.service - Daily apt download activities.
Sep 11 09:17:01 pve CRON[825838]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 09:17:01 pve CRON[825839]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 09:17:01 pve CRON[825838]: pam_unix(cron:session): session closed for user root
Sep 11 10:17:01 pve CRON[835226]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 10:17:01 pve CRON[835227]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 10:17:01 pve CRON[835226]: pam_unix(cron:session): session closed for user root
Sep 11 11:17:01 pve CRON[844588]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 11:17:01 pve CRON[844589]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 11:17:01 pve CRON[844588]: pam_unix(cron:session): session closed for user root
Sep 11 12:17:01 pve CRON[853946]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 12:17:01 pve CRON[853947]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 12:17:01 pve CRON[853946]: pam_unix(cron:session): session closed for user root


*** COMPUTER MANUALLY WOKE UP ***


Sep 11 12:54:30 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 11 12:54:30 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 11 12:54:30 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 11 12:54:34 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 11 12:54:34 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 11 12:54:34 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Sep 11 12:54:47 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Sep 11 12:54:47 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Sep 11 12:54:47 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Sep 11 12:54:48 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Sep 11 12:54:50 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 11 12:54:50 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 11 12:54:50 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 11 12:54:50 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 11 12:54:50 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 11 12:54:50 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Sep 11 13:17:01 pve CRON[863337]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 13:17:01 pve CRON[863338]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 13:17:01 pve CRON[863337]: pam_unix(cron:session): session closed for user root
Sep 11 14:17:01 pve CRON[872708]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 14:17:01 pve CRON[872709]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 14:17:01 pve CRON[872708]: pam_unix(cron:session): session closed for user root
Sep 11 14:33:40 pve pvedaemon[45932]: <root@pam> successful auth for user 'root@pam'
Server View

Points to note:

1. When the computer is manually put to sleep and manually awoken, only the adapter link-state changes are recorded.
2. When the computer automatically wakes up, a register dump is taken and the adapter is reset.

Now I recon it was at point (2.) that the kernal was crashing before (and I would certainly prefer an adapter reset than a kernel crash!) but there is still something weird (anomalous) going on when the computer wakes up automatically from sleep.

 

[tictag]

I have upgrade PVE to Linux 6.2.16-12-pve. All seems good so far. I will be testing point (2.) above over the next few days.

 

[tictag]

After upgrading to Linux 6.2.16-12-pve yesterday, my device crashed this morning the moment my laptop resumed from sleep, I would say this was the same problem that started this thread but this time, nothing was recorded in the SysLog:

Sep 17 14:26:02 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 14:26:02 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 17 14:26:02 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 17 14:26:06 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 14:26:06 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 17 14:26:06 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Sep 17 14:26:17 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Sep 17 14:26:17 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Sep 17 14:26:17 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Sep 17 14:26:18 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Sep 17 14:26:20 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 14:26:20 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 17 14:26:20 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Sep 17 14:26:21 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 17 14:26:21 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 17 14:26:21 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
-- Reboot --
Sep 17 14:35:32 pve kernel: Linux version 6.2.16-12-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-12 (2023-09-04T13:21Z) ()
Sep 17 14:35:32 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-12-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

How do I downgrade back to kernel Linux version 6.2.16-3-pve?

 

[tictag]

Sorry top be a bother but does anybody know how to downgrade the Proxmox Kernel?

 

[_gabriel]

to pin kernel :
https://pve.proxmox.com/wiki/Host_Bootloader

 

[tictag]

Ha! That was actually a lot easier than I thought it would be! Thank you, @_gabriel

I'm back on Linux kernel 6.2.16-3-pve and going to test this version for a week, then upgrade again and re-test for another week. This should provide the evidence needed to prove v6.2.16-12-pve broke what was previously fixed.

Are there any release notes for these kernel updates?
Is the code open source?

 

[tictag]

My NUC crashed this afternoon after my Windows laptop resumed from sleep, and this was on kernel v6.2.16-3-pve, which previously had worked flawlessly for over a week. I don't know, maybe it only crashes when it feels sad, as we all do from time to time.

Sep 23 15:12:08 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 23 15:12:08 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 23 15:12:08 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
Sep 23 15:12:12 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 23 15:12:12 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 23 15:12:12 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Sep 23 15:12:26 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Sep 23 15:12:26 pve kernel: vmbr0: port 3(enp5s0) entered disabled state
Sep 23 15:12:26 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Down
Sep 23 15:12:27 pve kernel: vmbr0: port 2(enp4s0) entered disabled state
Sep 23 15:12:29 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 23 15:12:29 pve kernel: vmbr0: port 3(enp5s0) entered blocking state
Sep 23 15:12:29 pve kernel: vmbr0: port 3(enp5s0) entered forwarding state
Sep 23 15:12:30 pve kernel: igc 0000:04:00.0 enp4s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Sep 23 15:12:30 pve kernel: vmbr0: port 2(enp4s0) entered blocking state
Sep 23 15:12:30 pve kernel: vmbr0: port 2(enp4s0) entered forwarding state
-- Reboot --
Sep 23 15:20:12 pve kernel: Linux version 6.2.16-3-pve (tom@sbuild) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) ()
Sep 23 15:20:12 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
Sep 23 15:20:12 pve kernel: KERNEL supported cpus:

 

[tictag]

Another crash today, same story, Windows laptop resumes from hibernation sleep, few minutes later everything stops working - same syslog as above.

I have upgraded to kernel v6.2.16-14-pve. Will test over the coming week.

 

[fweber]

Hi, thanks for sharing your observations! Sorry to hear the new kernel did not fix the issues.

I did find another kernel patch [1] which apparently fixes a race condition that produce a kernel panic when the link is taken down. This sounds like it could be related to the issue you're seeing and it being a race condition would explain why this only happens sporadically. But unfortunately this is only a guess at this point. The patch is included in the mainline kernel [2], but not in the current Ubuntu 6.2 kernel or our kernel.

It you're interested, it might be worthwhile to try whether the reproducer described in [1] provokes a crash -- if it does, it could be a hint that the patch [1] may solve the issue.

[1] https://lore.kernel.org/netdev/20230612205208.115292-2-anthony.l.nguyen@intel.com/
[2] https://git.kernel.org/pub/scm/linu...c?id=e43516f5978d11d36511ce63d31d1da4db916510