Kernel BUG CPU Soft lockup.(VM/HOST freezes)

[Stead]

I have no more softlock, but system becomes unresponsive, I have to restart it, no logs. No high IO aswell.
EDIT: It was in fact a softlock.

 

[spamsam]

I have no more softlock, but system becomes unresponsive, I have to restart it, no logs. No high IO aswell.

Do you by any chance have ceph installed on the host?

 

[Stead]

Ceph is not installed.

This happened this week-end, apparently, there was a softlock logged this time.



I downgraded back to 5.13.19-6, I will check if lock occurs.

 

[Stead]

In fact, it is still softlocking, I can't get a clue about what is going on.

 

[spamsam]

Oddly enough, it seems to have been one specific VM that triggered the crash for me. Granted it was a backup server. Yet, I have 2 other proxmox hosts that also have backup servers on them with no apparent issues.

 

[Stead]

For me no VM seems to trigger the crash, server crashed like 4 times in a row, with no load or IO.
I tried to downgrade to 5.11 kernel (I know I had no issue with proxmox 7.1, and it was the default kernel at this time)

 

[spamsam]

HW raid controller? any disks in raid? I got 3 x 1TB SSD in RAID5 (local-lvm) and 2 x 4TB HDDs in RAID1. Xeon E series.

 

[Stead]

No hardware raid.
I have only two ZFS pool. One for root with one NVMe.

And basically my setup :





Ram is 64GB ECC unregistered.
Since I downgraded to 5.11 it did not crashed while it was crashing several times a day on 5.15 and 5.19.

 

[Stead]

Little update, a SSD died, so I changed it, during resilvering, host crashed within seconds with kernel 5.15.
I tried ten times, a softlock occured everytime.

I pinned 5.11.22-7 kernel back, I was able to resilver without any issue.
Under 5.15 and 5.19 I also crashed without high IO, but it took minutes/hours.

I don't know it's related to file system, but it looks like so.
It would be nice to have a word from the Proxmox Staff, because I am not a kernel engineer,
I don't know where to look.

On Kernel 5.11 my uptime is 7 days, no crash so far.

 

[spamsam]

According to https://pve.proxmox.com/wiki/Roadmap.

intel_iommu defaulted to on in kernels 5.15 based kernels up to pve-kernel-5.15.35-3-pve with version 5.15.35-6. An enabled IOMMU can cause problems with older hardware - issues were reported with, for example, HP DL380 g8 servers, and Dell R610 servers (both older than 10 years).
The issue can be fixed by explicitly disabling intel_iommu on the kernel commandline (intel_iommu=off) following the reference documentation - https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline
As the setting has been reverted in newer pve-kernel-5.15 packages the issue is now mostly relevant during installation from the ISO.
Certain systems may need to explicitly enable iommu=pt (SR-IOV pass-through) on the kernel command line.
There are some reports for this to solve issues with Avago/LSI RAID controllers, for example in a Dell R340 Server booted in legacy mode.
Certain systems may need to work around a HW issue with TDP (two-dimensional paging), a now by default enabled technic to speed up VM to host, or vice versa, switches.
If you see a kernel log message containing KVM: entry failed, hardware error 0x80000021 you are affected.
This has been adressed with the kernel package pve-kernel-5.15.39-3-pve with version 5.15.39-3 and newer, which is available on all repositories.

Wondering if the issue is probably related to that.

 

[Stead]

I am using an AMD stack, no iommu is enabled on my server.
I tried to blacklist nouveau driver since it was a cause of softlock a few year ago.

I unpined my 5.11, went back on 5.15. After 15 minutes of idle server crashed :


Jan 07 10:37:24 raptor pvestatd[3268]: storage 'nas0' is not online
Jan 07 10:37:24 raptor pvestatd[3268]: status update time (5.092 seconds)
Jan 07 10:37:33 raptor kernel: ================================================================================
Jan 07 10:37:33 raptor kernel: UBSAN: array-index-out-of-bounds in kernel/sched/sched.h:2842:9
Jan 07 10:37:33 raptor kernel: index 474671110 is out of range for type 'long unsigned int [8192]'
Jan 07 10:37:33 raptor kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P O 5.15.83-1-pve #1
Jan 07 10:37:33 raptor kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./A520M-ITX/ac, BIOS P2.10 10/20/2022
Jan 07 10:37:33 raptor kernel: Call Trace:
Jan 07 10:37:33 raptor kernel: <IRQ>
Jan 07 10:37:33 raptor kernel: dump_stack_lvl+0x4a/0x63
Jan 07 10:37:33 raptor kernel: dump_stack+0x10/0x16
Jan 07 10:37:33 raptor kernel: ubsan_epilogue+0x9/0x49
Jan 07 10:37:33 raptor kernel: __ubsan_handle_out_of_bounds.cold+0x44/0x49
Jan 07 10:37:33 raptor kernel: ? sched_core_balance+0x1c0/0x390
Jan 07 10:37:33 raptor kernel: ? native_x2apic_icr_read+0x20/0x20
Jan 07 10:37:33 raptor kernel: update_blocked_averages+0x7b7/0x7d0
Jan 07 10:37:33 raptor kernel: ? native_x2apic_icr_read+0x20/0x20
Jan 07 10:37:33 raptor kernel: ? lapic_next_event+0x21/0x30
Jan 07 10:37:33 raptor kernel: ? clockevents_program_event+0xab/0x130
Jan 07 10:37:33 raptor kernel: run_rebalance_domains+0x4b/0x80
Jan 07 10:37:33 raptor kernel: __do_softirq+0xd9/0x2ea
Jan 07 10:37:33 raptor kernel: irq_exit_rcu+0x94/0xc0
Jan 07 10:37:33 raptor kernel: sysvec_apic_timer_interrupt+0x80/0x90
Jan 07 10:37:33 raptor kernel: </IRQ>
Jan 07 10:37:33 raptor kernel: <TASK>
Jan 07 10:37:33 raptor kernel: asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jan 07 10:37:33 raptor kernel: RIP: 0010:cpuidle_enter_state+0xd9/0x620
Jan 07 10:37:33 raptor kernel: Code: 3d f4 3f bf 52 e8 27 00 6e ff 49 89 c7 0f 1f 44 00 00 31 ff e8 68 0d 6e ff 80 7d d0 00 0f 85 5e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6a 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e5 03 00 00
Jan 07 10:37:33 raptor kernel: RSP: 0018:ffffbb96800ebe38 EFLAGS: 00000246
Jan 07 10:37:33 raptor kernel: RAX: ffff91ba3eaf0bc0 RBX: ffff91ab433d9c00 RCX: 0000000000000000
Jan 07 10:37:33 raptor kernel: RDX: 0000000000000119 RSI: 00000000239f5376 RDI: 0000000000000000
Jan 07 10:37:33 raptor kernel: RBP: ffffbb96800ebe88 R08: 00000229ddb5f42a R09: 0000000000008ca0
Jan 07 10:37:33 raptor kernel: R10: 0000000000000003 R11: 071c71c71c71c71c R12: ffffffffae8e6ec0
Jan 07 10:37:33 raptor kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 00000229ddb5f42a
Jan 07 10:37:33 raptor kernel: ? cpuidle_enter_state+0xc8/0x620
Jan 07 10:37:33 raptor kernel: cpuidle_enter+0x2e/0x50
Jan 07 10:37:33 raptor kernel: do_idle+0x20d/0x2b0
Jan 07 10:37:33 raptor kernel: cpu_startup_entry+0x20/0x30
Jan 07 10:37:33 raptor kernel: start_secondary+0x12a/0x180
Jan 07 10:37:33 raptor kernel: secondary_startup_64_no_verify+0xc2/0xcb
Jan 07 10:37:33 raptor kernel: </TASK>
Jan 07 10:37:33 raptor kernel: ================================================================================
Jan 07 10:37:33 raptor kernel: BUG: unable to handle page fault for address: 0000000090634b10
Jan 07 10:37:33 raptor kernel: #PF: supervisor read access in kernel mode
Jan 07 10:37:33 raptor kernel: #PF: error_code(0x0000) - not-present page
Jan 07 10:37:33 raptor kernel: PGD 0 P4D 0
Jan 07 10:37:33 raptor kernel: Oops: 0000 [#1] SMP NOPTI
Jan 07 10:37:33 raptor kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P O 5.15.83-1-pve #1
Jan 07 10:37:33 raptor kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./A520M-ITX/ac, BIOS P2.10 10/20/2022
Jan 07 10:37:33 raptor kernel: RIP: 0010:update_blocked_averages+0x691/0x7d0
Jan 07 10:37:33 raptor kernel: Code: 41 5c 41 5d 41 5e 41 5f 5d e9 3b 6c ef 00 4d 63 ac 24 e0 0a 00 00 48 c7 c3 98 04 02 00 49 81 fd ff 1f 00 00 0f 87 17 01 00 00 <4a> 03 1c ed e0 0a 0c ae 4c 8b 2b 4d 85 ed 0f 84 42 ff ff ff 41 83
Jan 07 10:37:33 raptor kernel: RSP: 0018:ffffbb9680270ef8 EFLAGS: 00010092
Jan 07 10:37:33 raptor kernel: RAX: ffff91ab40264b00 RBX: 0000000000020498 RCX: ffff91ba3eae0588
Jan 07 10:37:33 raptor kernel: RDX: 0000000000000000 RSI: 0000000000000027 RDI: ffff91ba3eae0580
Jan 07 10:37:33 raptor kernel: RBP: ffffbb9680270f58 R08: 0000000000000003 R09: 0000000000000001
Jan 07 10:37:33 raptor kernel: R10: ffff91ab4eec87d0 R11: ffffffffadca8c60 R12: ffffffffaca89270
Jan 07 10:37:33 raptor kernel: R13: 000000001c4ae806 R14: 0000000000000018 R15: ffff91ba3eaf1440
Jan 07 10:37:33 raptor kernel: FS: 0000000000000000(0000) GS:ffff91ba3eac0000(0000) knlGS:0000000000000000
Jan 07 10:37:33 raptor kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 07 10:37:33 raptor kernel: CR2: 0000000090634b10 CR3: 00000001107ee000 CR4: 0000000000350ee0
Jan 07 10:37:33 raptor kernel: Call Trace:
Jan 07 10:37:33 raptor kernel: <IRQ>
Jan 07 10:37:33 raptor kernel: ? native_x2apic_icr_read+0x20/0x20
Jan 07 10:37:33 raptor kernel: ? lapic_next_event+0x21/0x30
Jan 07 10:37:33 raptor kernel: ? clockevents_program_event+0xab/0x130
Jan 07 10:37:33 raptor kernel: run_rebalance_domains+0x4b/0x80
Jan 07 10:37:33 raptor kernel: __do_softirq+0xd9/0x2ea
Jan 07 10:37:33 raptor kernel: irq_exit_rcu+0x94/0xc0
Jan 07 10:37:33 raptor kernel: sysvec_apic_timer_interrupt+0x80/0x90
Jan 07 10:37:33 raptor kernel: </IRQ>
Jan 07 10:37:33 raptor kernel: <TASK>
Jan 07 10:37:33 raptor kernel: asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jan 07 10:37:33 raptor kernel: RIP: 0010:cpuidle_enter_state+0xd9/0x620
Jan 07 10:37:33 raptor kernel: Code: 3d f4 3f bf 52 e8 27 00 6e ff 49 89 c7 0f 1f 44 00 00 31 ff e8 68 0d 6e ff 80 7d d0 00 0f 85 5e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6a 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e5 03 00 00
Jan 07 10:37:33 raptor kernel: RSP: 0018:ffffbb96800ebe38 EFLAGS: 00000246
Jan 07 10:37:33 raptor kernel: RAX: ffff91ba3eaf0bc0 RBX: ffff91ab433d9c00 RCX: 0000000000000000
Jan 07 10:37:33 raptor kernel: RDX: 0000000000000119 RSI: 00000000239f5376 RDI: 0000000000000000
Jan 07 10:37:33 raptor kernel: RBP: ffffbb96800ebe88 R08: 00000229ddb5f42a R09: 0000000000008ca0
Jan 07 10:37:33 raptor kernel: R10: 0000000000000003 R11: 071c71c71c71c71c R12: ffffffffae8e6ec0
Jan 07 10:37:33 raptor kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 00000229ddb5f42a
Jan 07 10:37:33 raptor kernel: ? cpuidle_enter_state+0xc8/0x620
Jan 07 10:37:33 raptor kernel: cpuidle_enter+0x2e/0x50
Jan 07 10:37:33 raptor kernel: do_idle+0x20d/0x2b0
Jan 07 10:37:33 raptor kernel: cpu_startup_entry+0x20/0x30
Jan 07 10:37:33 raptor kernel: start_secondary+0x12a/0x180
Jan 07 10:37:33 raptor kernel: secondary_startup_64_no_verify+0xc2/0xcb
Jan 07 10:37:33 raptor kernel: </TASK>
Jan 07 10:37:33 raptor kernel: Modules linked in: nft_counter nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel binfmt_misc tcp_diag inet_diag cmac nls_utf8 cifs cifs_arc4 cifs_md4 fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common amd64_edac iwlmvm edac_mce_amd mac80211 snd_hda_codec_realtek kvm_amd snd_hda_codec_generic ledtrig_audio libarc4 snd_hda_codec_hdmi kvm snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi irqbypass snd_hda_codec btusb crct10dif_pclmul ghash_clmulni_intel btrtl btbcm btintel aesni_intel snd_hda_core snd_hwdep crypto_simd bluetooth snd_pcm cryptd rapl joydev ecdh_generic cdc_acm snd_timer ecc wmi_bmof iwlwifi pcspkr snd efi_pstore k10temp cfg80211
Jan 07 10:37:33 raptor kernel: ccp soundcore mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb hid_generic usbhid hid xhci_pci crc32_pclmul xhci_pci_renesas i2c_piix4 r8169 nvme ahci realtek xhci_hcd libahci nvme_core wmi gpio_amdpt gpio_generic
Jan 07 10:37:33 raptor kernel: CR2: 0000000090634b10
Jan 07 10:37:33 raptor kernel: ---[ end trace 474280cf0621cc58 ]---
Jan 07 10:37:33 raptor kernel: NMI watchdog: Watchdog detected hard LOCKUP on cpu 7
Jan 07 10:37:33 raptor kernel: Modules linked in: nft_counter nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel binfmt_misc tcp_diag inet_diag cmac nls_utf8 cifs cifs_arc4 cifs_md4 fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common amd64_edac iwlmvm edac_mce_amd mac80211 snd_hda_codec_realtek kvm_amd snd_hda_codec_generic ledtrig_audio libarc4 snd_hda_codec_hdmi kvm snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi irqbypass snd_hda_codec btusb crct10dif_pclmul ghash_clmulni_intel btrtl btbcm btintel aesni_intel snd_hda_core snd_hwdep crypto_simd bluetooth snd_pcm cryptd rapl joydev ecdh_generic cdc_acm snd_timer ecc wmi_bmof iwlwifi pcspkr snd efi_pstore k10temp cfg80211
Jan 07 10:37:33 raptor kernel: ccp soundcore mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb hid_generic usbhid hid xhci_pci crc32_pclmul xhci_pci_renesas i2c_piix4 r8169 nvme ahci realtek xhci_hcd libahci nvme_core wmi gpio_amdpt gpio_generic
Jan 07 10:37:33 raptor kernel: CPU: 7 PID: 3549 Comm: CPU 2/KVM Tainted: P D O 5.15.83-1-pve #1
Jan 07 10:37:33 raptor kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./A520M-ITX/ac, BIOS P2.10 10/20/2022
Jan 07 10:37:33 raptor kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x14d/0x240
Jan 07 10:37:33 raptor kernel: Code: 5d e9 97 c3 ec 00 83 c6 01 c1 e2 10 c1 e6 12 09 f2 41 89 d7 89 d0 c1 e8 10 66 87 43 02 89 c1 c1 e1 10 75 6a 31 c9 eb 02 f3 90 <8b> 03 66 85 c0 75 f7 89 c6 66 31 f6 41 39 f7 0f 84 ac 00 00 00 c6
Jan 07 10:37:33 raptor kernel: RSP: 0018:ffffbb969137bb60 EFLAGS: 00000002
Jan 07 10:37:33 raptor kernel: RAX: 0000000000280101 RBX: ffff91ba3eaf0bc0 RCX: 0000000000000000
Jan 07 10:37:33 raptor kernel: RDX: 0000000000200000 RSI: 0000000000200000 RDI: ffff91ba3eaf0bc0
Jan 07 10:37:33 raptor kernel: RBP: ffffbb969137bb88 R08: 0000000000000002 R09: 0000000000000bb8
Jan 07 10:37:33 raptor kernel: R10: 0000000000000bb8 R11: 0000000000000000 R12: ffff91ba3ebf1940
Jan 07 10:37:33 raptor kernel: R13: ffff91ba3ebf0bc0 R14: ffff91ba3eaf0bc0 R15: 0000000000200000
Jan 07 10:37:33 raptor kernel: FS: 00007f0c78bff700(0000) GS:ffff91ba3ebc0000(0000) knlGS:0000000000000000
Jan 07 10:37:33 raptor kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 07 10:37:33 raptor kernel: CR2: 000001dcb4a88590 CR3: 00000001107ee000 CR4: 0000000000350ee0
Jan 07 10:37:33 raptor kernel: Call Trace:
Jan 07 10:37:33 raptor kernel: <TASK>
Jan 07 10:37:33 raptor kernel: _raw_spin_lock+0x22/0x30
Jan 07 10:37:33 raptor kernel: raw_spin_rq_lock_nested+0x17/0x80
Jan 07 10:37:33 raptor kernel: double_rq_lock+0x8e/0xb0
Jan 07 10:37:33 raptor kernel: yield_to+0xc8/0x270
Jan 07 10:37:33 raptor kernel: kvm_vcpu_yield_to+0x3a/0xb0 [kvm]
Jan 07 10:37:33 raptor kernel: kvm_vcpu_on_spin+0x153/0x2a0 [kvm]
Jan 07 10:37:33 raptor kernel: pause_interception+0x7a/0x140 [kvm_amd]
Jan 07 10:37:33 raptor kernel: ? svm_queue_exception+0x130/0x130 [kvm_amd]
Jan 07 10:37:33 raptor kernel: svm_invoke_exit_handler+0x74/0x1b0 [kvm_amd]
Jan 07 10:37:33 raptor kernel: handle_exit+0x98/0x230 [kvm_amd]
Jan 07 10:37:33 raptor kernel: kvm_arch_vcpu_ioctl_run+0xdd6/0x1730 [kvm]
Jan 07 10:37:33 raptor kernel: ? __wake_up_common+0x7e/0x140
Jan 07 10:37:33 raptor kernel: ? __wake_up_locked_key+0x1b/0x30
Jan 07 10:37:33 raptor kernel: ? eventfd_write+0xdb/0x270
Jan 07 10:37:33 raptor kernel: kvm_vcpu_ioctl+0x252/0x6b0 [kvm]
Jan 07 10:37:33 raptor kernel: ? vfs_write+0xcb/0x270
Jan 07 10:37:33 raptor kernel: ? __fget_files+0x86/0xc0
Jan 07 10:37:33 raptor kernel: __x64_sys_ioctl+0x95/0xd0
Jan 07 10:37:33 raptor kernel: do_syscall_64+0x5c/0xc0
Jan 07 10:37:33 raptor kernel: ? do_syscall_64+0x69/0xc0
Jan 07 10:37:33 raptor kernel: ? __x64_sys_write+0x1a/0x20
Jan 07 10:37:33 raptor kernel: ? do_syscall_64+0x69/0xc0
Jan 07 10:37:33 raptor kernel: ? do_syscall_64+0x69/0xc0
Jan 07 10:37:33 raptor kernel: ? do_syscall_64+0x69/0xc0
Jan 07 10:37:33 raptor kernel: ? do_syscall_64+0x69/0xc0
Jan 07 10:37:33 raptor kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Jan 07 10:37:33 raptor kernel: RIP: 0033:0x7f0c86a535f7
Jan 07 10:37:33 raptor kernel: Code: 00 00 00 48 8b 05 99 c8 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 69 c8 0d 00 f7 d8 64 89 01 48
Jan 07 10:37:33 raptor kernel: RSP: 002b:00007f0c78bfa288 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Jan 07 10:37:33 raptor kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f0c86a535f7
Jan 07 10:37:33 raptor kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000028
Jan 07 10:37:33 raptor kernel: RBP: 000055bf07a3d180 R08: 000055bf0654b250 R09: 00000000ffffffff
Jan 07 10:37:33 raptor kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
Jan 07 10:37:33 raptor kernel: R13: 000055bf06c30140 R14: 0000000000000000 R15: 0000000000000000
Jan 07 10:37:33 raptor kernel: </TASK>
-- Reboot --


I pinned back 5.11.

 

[Stead]

Another kind of softlock:

Jan 07 10:37:35 raptor pvestatd[3268]: storage 'nas0' is not online
Jan 07 10:37:35 raptor pvestatd[3268]: status update time (5.089 seconds)
Jan 07 10:37:44 raptor pvestatd[3268]: storage 'nas0' is not online
Jan 07 10:38:01 raptor kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Jan 07 10:38:01 raptor kernel: NMI watchdog: Watchdog detected hard LOCKUP on cpu 5
Jan 07 10:38:01 raptor kernel: Modules linked in: nft_counter nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel binfmt_misc tcp_diag inet_diag cmac nls_utf8 cifs cifs_arc4 cifs_md4 fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common amd64_edac iwlmvm edac_mce_amd mac80211 snd_hda_codec_realtek kvm_amd snd_hda_codec_generic ledtrig_audio libarc4 snd_hda_codec_hdmi kvm snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi irqbypass snd_hda_codec btusb crct10dif_pclmul ghash_clmulni_intel btrtl btbcm btintel aesni_intel snd_hda_core snd_hwdep crypto_simd bluetooth snd_pcm cryptd rapl joydev ecdh_generic cdc_acm snd_timer ecc wmi_bmof iwlwifi pcspkr snd efi_pstore k10temp cfg80211
Jan 07 10:38:01 raptor kernel: ccp soundcore mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb hid_generic usbhid hid xhci_pci crc32_pclmul xhci_pci_renesas i2c_piix4 r8169 nvme ahci realtek xhci_hcd libahci nvme_core wmi gpio_amdpt gpio_generic
Jan 07 10:38:01 raptor kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: P D O L 5.15.83-1-pve #1
Jan 07 10:38:01 raptor kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./A520M-ITX/ac, BIOS P2.10 10/20/2022
Jan 07 10:38:01 raptor kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x79/0x240
Jan 07 10:38:01 raptor kernel: Code: 2b 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 03 30 e4 09 d0 a9 00 01 ff ff 0f 85 13 01 00 00 85 c0 74 0e 8b 03 84 c0 74 08 f3 90 <8b> 03 84 c0 75 f8 b8 01 00 00 00 66 89 03 5b 41 5c 41 5d 41 5e 41
Jan 07 10:38:01 raptor kernel: RSP: 0018:ffffbb96802d8df0 EFLAGS: 00000002
Jan 07 10:38:01 raptor kernel: RAX: 0000000000000101 RBX: ffffffffae77fdc0 RCX: 0000000000000000
Jan 07 10:38:01 raptor kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffae77fdc0
Jan 07 10:38:01 raptor kernel: RBP: ffffbb96802d8e18 R08: ffffffffae77fdc0 R09: 000000000001be49
Jan 07 10:38:01 raptor kernel: R10: 80000001000828c4 R11: ffffffffadca8c60 R12: 0000000000000087
Jan 07 10:38:01 raptor kernel: R13: 00000000000319c0 R14: ffffffffae77fdc0 R15: 000000010007ee2b
Jan 07 10:38:01 raptor kernel: FS: 0000000000000000(0000) GS:ffff91ba3eb40000(0000) knlGS:0000000000000000
Jan 07 10:38:01 raptor kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 07 10:38:01 raptor kernel: CR2: 00007f80d4150aa0 CR3: 00000001725e0000 CR4: 0000000000350ee0
Jan 07 10:38:01 raptor kernel: Call Trace:
Jan 07 10:38:01 raptor kernel: <IRQ>
Jan 07 10:38:01 raptor kernel: _raw_spin_lock_irqsave+0x3d/0x50
Jan 07 10:38:01 raptor kernel: rcu_sched_clock_irq+0x5c2/0x8a0
Jan 07 10:38:01 raptor kernel: update_process_times+0x94/0xd0
Jan 07 10:38:01 raptor kernel: ? tick_do_update_jiffies64.part.0+0xa0/0xa0
Jan 07 10:38:01 raptor kernel: tick_sched_handle+0x29/0x70
Jan 07 10:38:01 raptor kernel: ? tick_do_update_jiffies64.part.0+0xa0/0xa0
Jan 07 10:38:01 raptor kernel: tick_sched_timer+0x8c/0xf0
Jan 07 10:38:01 raptor kernel: __hrtimer_run_queues+0x109/0x210
Jan 07 10:38:01 raptor kernel: hrtimer_interrupt+0x101/0x230
Jan 07 10:38:01 raptor kernel: __sysvec_apic_timer_interrupt+0x61/0xe0
Jan 07 10:38:01 raptor kernel: sysvec_apic_timer_interrupt+0x7b/0x90
Jan 07 10:38:01 raptor kernel: </IRQ>
Jan 07 10:38:01 raptor kernel: <TASK>
Jan 07 10:38:01 raptor kernel: asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jan 07 10:38:01 raptor kernel: RIP: 0010:cpuidle_enter_state+0xd9/0x620
Jan 07 10:38:01 raptor kernel: Code: 3d f4 3f bf 52 e8 27 00 6e ff 49 89 c7 0f 1f 44 00 00 31 ff e8 68 0d 6e ff 80 7d d0 00 0f 85 5e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6a 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e5 03 00 00
Jan 07 10:38:01 raptor kernel: RSP: 0018:ffffbb96800fbe38 EFLAGS: 00000246
Jan 07 10:38:01 raptor kernel: RAX: ffff91ba3eb70bc0 RBX: ffff91ab433dd000 RCX: 0000000000000000
Jan 07 10:38:01 raptor kernel: RDX: 0000000000000120 RSI: 00000000239f5376 RDI: 0000000000000000
Jan 07 10:38:01 raptor kernel: RBP: ffffbb96800fbe88 R08: 00000237d6324066 R09: 0000000000008ca0
Jan 07 10:38:01 raptor kernel: R10: 0000000000000003 R11: 071c71c71c71c71c R12: ffffffffae8e6ec0
Jan 07 10:38:01 raptor kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 00000237d6324066
Jan 07 10:38:01 raptor kernel: ? cpuidle_enter_state+0xc8/0x620
Jan 07 10:38:01 raptor kernel: cpuidle_enter+0x2e/0x50
Jan 07 10:38:01 raptor kernel: do_idle+0x20d/0x2b0
Jan 07 10:38:01 raptor kernel: cpu_startup_entry+0x20/0x30
Jan 07 10:38:01 raptor kernel: start_secondary+0x12a/0x180
Jan 07 10:38:01 raptor kernel: secondary_startup_64_no_verify+0xc2/0xcb
Jan 07 10:38:01 raptor kernel: </TASK>
Jan 07 10:38:01 raptor kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 49s! [kworker/0:2:88]
Jan 07 10:38:01 raptor kernel: Modules linked in: nft_counter nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel binfmt_misc tcp_diag inet_diag cmac nls_utf8 cifs cifs_arc4 cifs_md4 fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common amd64_edac iwlmvm edac_mce_amd mac80211 snd_hda_codec_realtek kvm_amd snd_hda_codec_generic ledtrig_audio libarc4 snd_hda_codec_hdmi kvm snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi irqbypass snd_hda_codec btusb crct10dif_pclmul ghash_clmulni_intel btrtl btbcm btintel aesni_intel snd_hda_core snd_hwdep crypto_simd bluetooth snd_pcm cryptd rapl joydev ecdh_generic cdc_acm snd_timer ecc wmi_bmof iwlwifi pcspkr snd efi_pstore k10temp cfg80211
Jan 07 10:38:01 raptor kernel: ccp soundcore mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb hid_generic usbhid hid xhci_pci crc32_pclmul xhci_pci_renesas i2c_piix4 r8169 nvme ahci realtek xhci_hcd libahci nvme_core wmi gpio_amdpt gpio_generic
Jan 07 10:38:01 raptor kernel: CPU: 0 PID: 88 Comm: kworker/0:2 Tainted: P D O L 5.15.83-1-pve #1
Jan 07 10:38:01 raptor kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./A520M-ITX/ac, BIOS P2.10 10/20/2022
Jan 07 10:38:01 raptor kernel: Workqueue: events netstamp_clear
Jan 07 10:38:01 raptor kernel: RIP: 0010:smp_call_function_many_cond+0x13c/0x360
Jan 07 10:38:01 raptor kernel: Code: 01 41 89 c4 73 2d 4d 63 ec 48 8b 13 49 81 fd ff 1f 00 00 0f 87 e3 01 00 00 4a 03 14 ed e0 0a 0c ae 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb bc 48 83 c4 40 5b 41 5c 41 5d 41 5e 41 5f
Jan 07 10:38:01 raptor kernel: RSP: 0018:ffffbb968042fcf0 EFLAGS: 00000202
Jan 07 10:38:01 raptor kernel: RAX: 0000000000000011 RBX: ffff91ba3ea31c00 RCX: 0000000000000003
Jan 07 10:38:01 raptor kernel: RDX: ffff91ba3eaf5f60 RSI: 0000000000000000 RDI: ffff91ab40067cb8
Jan 07 10:38:01 raptor kernel: RBP: ffffbb968042fd58 R08: 0000000000000000 R09: 0000000000000000
Jan 07 10:38:01 raptor kernel: R10: 0000000000000003 R11: fffffffffffffff8 R12: 0000000000000003
Jan 07 10:38:01 raptor kernel: R13: 0000000000000003 R14: 0000000000000001 R15: 0000000000000020
Jan 07 10:38:01 raptor kernel: FS: 0000000000000000(0000) GS:ffff91ba3ea00000(0000) knlGS:0000000000000000
Jan 07 10:38:01 raptor kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 07 10:38:01 raptor kernel: CR2: 0000047422a3a040 CR3: 0000000aa4e10000 CR4: 0000000000350ef0
Jan 07 10:38:01 raptor kernel: Call Trace:
Jan 07 10:38:01 raptor kernel: <TASK>
Jan 07 10:38:01 raptor kernel: ? text_poke_loc_init+0x190/0x190
Jan 07 10:38:01 raptor kernel: on_each_cpu_cond_mask+0x22/0x30
Jan 07 10:38:01 raptor kernel: text_poke_bp_batch+0xb2/0x270
Jan 07 10:38:01 raptor kernel: text_poke_finish+0x1f/0x40
Jan 07 10:38:01 raptor kernel: arch_jump_label_transform_apply+0x1a/0x30
Jan 07 10:38:01 raptor kernel: __jump_label_update+0xf3/0x140
Jan 07 10:38:01 raptor kernel: jump_label_update+0xba/0xe0
Jan 07 10:38:01 raptor kernel: static_key_enable_cpuslocked+0x77/0xa0
Jan 07 10:38:01 raptor kernel: static_key_enable+0x1b/0x30
Jan 07 10:38:01 raptor kernel: netstamp_clear+0x2d/0x40
Jan 07 10:38:01 raptor kernel: process_one_work+0x22b/0x3d0
Jan 07 10:38:01 raptor kernel: worker_thread+0x53/0x420
Jan 07 10:38:01 raptor kernel: ? process_one_work+0x3d0/0x3d0
Jan 07 10:38:01 raptor kernel: kthread+0x12a/0x150
Jan 07 10:38:01 raptor kernel: ? set_kthread_struct+0x50/0x50
Jan 07 10:38:01 raptor kernel: ret_from_fork+0x22/0x30
Jan 07 10:38:01 raptor kernel: </TASK>
Jan 07 10:38:29 raptor kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 75s! [kworker/0:2:88]