Jan 17 06:31:51 pmox kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Jan 17 06:31:51 pmox kernel: rcu: 1-...!: (1 GPs behind) idle=b680/0/0x0 softirq=59616/59616 fqs=1 (false positive?)
Jan 17 06:31:51 pmox kernel: rcu: 6-...!: (2 GPs behind) idle=4b00/0/0x0 softirq=254744/254746 fqs=1 (false positive?)
Jan 17 06:31:51 pmox kernel: rcu: 7-...!: (23 GPs behind) idle=72e0/0/0x0 softirq=54375/54375 fqs=1 (false positive?)
Jan 17 06:31:51 pmox kernel: rcu: 15-...!: (9 GPs behind) idle=76b0/0/0x0 softirq=86863/86863 fqs=1 (false positive?)
Jan 17 06:31:51 pmox kernel: rcu: (detected by 3, t=60004 jiffies, g=1524581, q=3919 ncpus=16)
Jan 17 06:31:51 pmox kernel: Sending NMI from CPU 3 to CPUs 1:
Jan 17 06:31:51 pmox kernel: nmi_backtrace_stall_check: CPU 1: NMIs are not reaching exc_nmi() handler, last activity: 1795203 jiffies ago.
Jan 17 06:31:51 pmox kernel: Sending NMI from CPU 3 to CPUs 6:
Jan 17 06:31:51 pmox kernel: nmi_backtrace_stall_check: CPU 6: NMIs are not reaching exc_nmi() handler, last activity: 1089575 jiffies ago.
Jan 17 06:31:51 pmox kernel: Sending NMI from CPU 3 to CPUs 7:
Jan 17 06:31:51 pmox kernel: nmi_backtrace_stall_check: CPU 7: NMIs are not reaching exc_nmi() handler, last activity: 9118767 jiffies ago.
Jan 17 06:31:51 pmox kernel: Sending NMI from CPU 3 to CPUs 15:
Jan 17 06:31:51 pmox kernel: nmi_backtrace_stall_check: CPU 15: NMIs are not reaching exc_nmi() handler, last activity: 712502 jiffies ago.
Jan 17 06:31:51 pmox kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 100007 jiffies! g1524581 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
Jan 17 06:31:51 pmox kernel: rcu: Possible timer handling issue on cpu=0 timer-softirq=193198
Jan 17 06:31:51 pmox kernel: rcu: rcu_preempt kthread starved for 100014 jiffies! g1524581 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
Jan 17 06:31:51 pmox kernel: rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Jan 17 06:31:51 pmox kernel: rcu: RCU grace-period kthread stack dump:
Jan 17 06:31:51 pmox kernel: task:rcu_preempt state:I stack:0 pid:17 tgid:17 ppid:2 flags:0x00004000
Jan 17 06:31:51 pmox kernel: Call Trace:
Jan 17 06:31:51 pmox kernel: <TASK>
Jan 17 06:31:51 pmox kernel: __schedule+0x401/0x15e0
Jan 17 06:31:51 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:31:51 pmox kernel: schedule+0x33/0x110
Jan 17 06:31:51 pmox kernel: schedule_timeout+0x95/0x170
Jan 17 06:31:51 pmox kernel: ? __pfx_process_timeout+0x10/0x10
Jan 17 06:31:51 pmox kernel: rcu_gp_fqs_loop+0x13d/0x550
Jan 17 06:31:51 pmox kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Jan 17 06:31:51 pmox kernel: rcu_gp_kthread+0xd7/0x1a0
Jan 17 06:31:51 pmox kernel: kthread+0xf2/0x120
Jan 17 06:31:51 pmox kernel: ? __pfx_kthread+0x10/0x10
Jan 17 06:31:51 pmox kernel: ret_from_fork+0x47/0x70
Jan 17 06:31:51 pmox kernel: ? __pfx_kthread+0x10/0x10
Jan 17 06:31:51 pmox kernel: ret_from_fork_asm+0x1b/0x30
Jan 17 06:31:51 pmox kernel: </TASK>
Jan 17 06:31:51 pmox kernel: rcu: Stack dump where RCU GP kthread last ran:
Jan 17 06:31:51 pmox kernel: Sending NMI from CPU 3 to CPUs 0:
Jan 17 06:31:51 pmox kernel: nmi_backtrace_stall_check: CPU 0: NMIs are not reaching exc_nmi() handler, last activity: 883745 jiffies ago.
Jan 17 06:33:53 pmox kernel: INFO: task khugepaged:123 blocked for more than 121 seconds.
Jan 17 06:33:53 pmox kernel: Tainted: P O 6.8.12-5-pve #1
Jan 17 06:33:53 pmox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 17 06:33:53 pmox kernel: task:khugepaged state:D stack:0 pid:123 tgid:123 ppid:2 flags:0x00004000
Jan 17 06:33:53 pmox kernel: Call Trace:
Jan 17 06:33:53 pmox kernel: <TASK>
Jan 17 06:33:53 pmox kernel: __schedule+0x401/0x15e0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: schedule+0x33/0x110
Jan 17 06:33:53 pmox kernel: schedule_timeout+0x157/0x170
Jan 17 06:33:53 pmox kernel: wait_for_completion+0x88/0x150
Jan 17 06:33:53 pmox kernel: __flush_work.isra.0+0x1b0/0x2d0
Jan 17 06:33:53 pmox kernel: ? __pfx_wq_barrier_func+0x10/0x10
Jan 17 06:33:53 pmox kernel: flush_work+0xe/0x20
Jan 17 06:33:53 pmox kernel: __lru_add_drain_all+0x14e/0x210
Jan 17 06:33:53 pmox kernel: lru_add_drain_all+0x10/0x20
Jan 17 06:33:53 pmox kernel: khugepaged+0x6e/0x9d0
Jan 17 06:33:53 pmox kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Jan 17 06:33:53 pmox kernel: ? __pfx_khugepaged+0x10/0x10
Jan 17 06:33:53 pmox kernel: kthread+0xf2/0x120
Jan 17 06:33:53 pmox kernel: ? __pfx_kthread+0x10/0x10
Jan 17 06:33:53 pmox kernel: ret_from_fork+0x47/0x70
Jan 17 06:33:53 pmox kernel: ? __pfx_kthread+0x10/0x10
Jan 17 06:33:53 pmox kernel: ret_from_fork_asm+0x1b/0x30
Jan 17 06:33:53 pmox kernel: </TASK>
Jan 17 06:33:53 pmox kernel: INFO: task nft:124678 blocked for more than 121 seconds.
Jan 17 06:33:53 pmox kernel: Tainted: P O 6.8.12-5-pve #1
Jan 17 06:33:53 pmox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 17 06:33:53 pmox kernel: task:nft state:D stack:0 pid:124678 tgid:124678 ppid:2035 flags:0x00000002
Jan 17 06:33:53 pmox kernel: Call Trace:
Jan 17 06:33:53 pmox kernel: <TASK>
Jan 17 06:33:53 pmox kernel: __schedule+0x401/0x15e0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: schedule+0x33/0x110
Jan 17 06:33:53 pmox kernel: schedule_timeout+0x157/0x170
Jan 17 06:33:53 pmox kernel: wait_for_completion+0x88/0x150
Jan 17 06:33:53 pmox kernel: __wait_rcu_gp+0x150/0x160
Jan 17 06:33:53 pmox kernel: synchronize_rcu+0x12d/0x140
Jan 17 06:33:53 pmox kernel: ? __pfx_call_rcu_hurry+0x10/0x10
Jan 17 06:33:53 pmox kernel: ? __pfx_wakeme_after_rcu+0x10/0x10
Jan 17 06:33:53 pmox kernel: nf_tables_abort+0x30b/0xee0 [nf_tables]
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? kmalloc_trace+0x139/0x360
Jan 17 06:33:53 pmox kernel: nfnetlink_rcv_batch+0x401/0xa90 [nfnetlink]
Jan 17 06:33:53 pmox kernel: nfnetlink_rcv+0x13e/0x180 [nfnetlink]
Jan 17 06:33:53 pmox kernel: netlink_unicast+0x1b3/0x2a0
Jan 17 06:33:53 pmox kernel: netlink_sendmsg+0x214/0x470
Jan 17 06:33:53 pmox kernel: ____sys_sendmsg+0x3f7/0x430
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ___sys_sendmsg+0x9a/0xf0
Jan 17 06:33:53 pmox kernel: __sys_sendmsg+0x89/0xf0
Jan 17 06:33:53 pmox kernel: __x64_sys_sendmsg+0x1d/0x30
Jan 17 06:33:53 pmox kernel: x64_sys_call+0xce5/0x2480
Jan 17 06:33:53 pmox kernel: do_syscall_64+0x81/0x170
Jan 17 06:33:53 pmox kernel: ? count_memcg_events.constprop.0+0x2a/0x50
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? handle_mm_fault+0xad/0x380
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? do_user_addr_fault+0x33e/0x660
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? irqentry_exit_to_user_mode+0x7b/0x260
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? irqentry_exit+0x43/0x50
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? exc_page_fault+0x94/0x1b0
Jan 17 06:33:53 pmox kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80
Jan 17 06:33:53 pmox kernel: RIP: 0033:0x72fbe7479b80
Jan 17 06:33:53 pmox kernel: RSP: 002b:00007ffd6518cd58 EFLAGS: 00000202 ORIG_RAX: 000000000000002e
Jan 17 06:33:53 pmox kernel: RAX: ffffffffffffffda RBX: 00007ffd6519df50 RCX: 000072fbe7479b80
Jan 17 06:33:53 pmox kernel: RDX: 0000000000000000 RSI: 00007ffd6519de00 RDI: 0000000000000003
Jan 17 06:33:53 pmox kernel: RBP: 00007ffd6519df00 R08: 00007ffd6518cd34 R09: 00005df66db48200
Jan 17 06:33:53 pmox kernel: R10: 000072fbe7661f00 R11: 0000000000000202 R12: 00005df66db47b50
Jan 17 06:33:53 pmox kernel: R13: 0000000000000400 R14: 00007ffd6518cd70 R15: 0000000000000001
Jan 17 06:33:53 pmox kernel: </TASK>
Jan 17 06:33:53 pmox kernel: INFO: task vgs:124685 blocked for more than 121 seconds.
Jan 17 06:33:53 pmox kernel: Tainted: P O 6.8.12-5-pve #1
Jan 17 06:33:53 pmox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 17 06:33:53 pmox kernel: task:vgs state:D stack:0 pid:124685 tgid:124685 ppid:2056 flags:0x00000002
Jan 17 06:33:53 pmox kernel: Call Trace:
Jan 17 06:33:53 pmox kernel: <TASK>
Jan 17 06:33:53 pmox kernel: __schedule+0x401/0x15e0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? __call_rcu_common+0xf9/0x750
Jan 17 06:33:53 pmox kernel: schedule+0x33/0x110
Jan 17 06:33:53 pmox kernel: schedule_timeout+0x157/0x170
Jan 17 06:33:53 pmox kernel: wait_for_completion+0x88/0x150
Jan 17 06:33:53 pmox kernel: __x64_sys_io_destroy+0xe0/0x130
Jan 17 06:33:53 pmox kernel: x64_sys_call+0x1692/0x2480
Jan 17 06:33:53 pmox kernel: do_syscall_64+0x81/0x170
Jan 17 06:33:53 pmox kernel: ? __fput+0x15e/0x2e0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? kmem_cache_free+0x331/0x400
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? __fput+0x15e/0x2e0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? syscall_exit_to_user_mode+0x86/0x260
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? do_syscall_64+0x8d/0x170
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? syscall_exit_to_user_mode+0x86/0x260
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? do_syscall_64+0x8d/0x170
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? do_syscall_64+0x8d/0x170
Jan 17 06:33:53 pmox kernel: ? syscall_exit_to_user_mode+0x86/0x260
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? do_syscall_64+0x8d/0x170
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? exc_page_fault+0x94/0x1b0
Jan 17 06:33:53 pmox kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80
Jan 17 06:33:53 pmox kernel: RIP: 0033:0x7746ac8e4799
Jan 17 06:33:53 pmox kernel: RSP: 002b:00007fff0470eb38 EFLAGS: 00000246 ORIG_RAX: 00000000000000cf
Jan 17 06:33:53 pmox kernel: RAX: ffffffffffffffda RBX: 00007746ac3f2b10 RCX: 00007746ac8e4799
Jan 17 06:33:53 pmox kernel: RDX: 00000000000099a1 RSI: 000058ada869ba70 RDI: 00007746acc72000
Jan 17 06:33:53 pmox kernel: RBP: 00007746acc72000 R08: 000058ada8680a50 R09: 00007fff0470edf8
Jan 17 06:33:53 pmox kernel: R10: 0000000000000007 R11: 0000000000000246 R12: 0000000000000000
Jan 17 06:33:53 pmox kernel: R13: 00007fff0470ee50 R14: 000058ad937795f0 R15: 00007746accab020
Jan 17 06:33:53 pmox kernel: </TASK>
Jan 17 06:33:53 pmox kernel: INFO: task ipset:124694 blocked for more than 121 seconds.
Jan 17 06:33:53 pmox kernel: Tainted: P O 6.8.12-5-pve #1
Jan 17 06:33:53 pmox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 17 06:33:53 pmox kernel: task:ipset state:D stack:0 pid:124694 tgid:124694 ppid:2046 flags:0x00000002
Jan 17 06:33:53 pmox kernel: Call Trace:
Jan 17 06:33:53 pmox kernel: <TASK>
Jan 17 06:33:53 pmox kernel: __schedule+0x401/0x15e0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? aa_sk_perm+0x46/0x250
Jan 17 06:33:53 pmox kernel: schedule+0x33/0x110
Jan 17 06:33:53 pmox kernel: schedule_preempt_disabled+0x15/0x30
Jan 17 06:33:53 pmox kernel: __mutex_lock.constprop.0+0x3f8/0x7a0
Jan 17 06:33:53 pmox kernel: ? ____sys_recvmsg+0x111/0x230
Jan 17 06:33:53 pmox kernel: __mutex_lock_slowpath+0x13/0x20
Jan 17 06:33:53 pmox kernel: mutex_lock+0x3c/0x50
Jan 17 06:33:53 pmox kernel: nft_rcv_nl_event+0xa5/0x1a0 [nf_tables]
Jan 17 06:33:53 pmox kernel: notifier_call_chain+0x61/0xe0
Jan 17 06:33:53 pmox kernel: blocking_notifier_call_chain+0x41/0x70
Jan 17 06:33:53 pmox kernel: netlink_release+0x6a0/0x710
Jan 17 06:33:53 pmox kernel: __sock_release+0x40/0xc0
Jan 17 06:33:53 pmox kernel: sock_close+0x15/0x30
Jan 17 06:33:53 pmox kernel: __fput+0xa1/0x2e0
Jan 17 06:33:53 pmox kernel: __fput_sync+0x1c/0x30
Jan 17 06:33:53 pmox kernel: __x64_sys_close+0x3e/0x90
Jan 17 06:33:53 pmox kernel: x64_sys_call+0x19ad/0x2480
Jan 17 06:33:53 pmox kernel: do_syscall_64+0x81/0x170
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? __sys_recvmsg+0x82/0xe0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? syscall_exit_to_user_mode+0x86/0x260
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? do_syscall_64+0x8d/0x170
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? do_fault+0x295/0x4c0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? __handle_mm_fault+0x88f/0xf20
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? __count_memcg_events+0x6f/0xe0
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? count_memcg_events.constprop.0+0x2a/0x50
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? handle_mm_fault+0xad/0x380
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? do_user_addr_fault+0x21e/0x660
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? irqentry_exit_to_user_mode+0x7b/0x260
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? irqentry_exit+0x43/0x50
Jan 17 06:33:53 pmox kernel: ? srso_return_thunk+0x5/0x5f
Jan 17 06:33:53 pmox kernel: ? exc_page_fault+0x94/0x1b0
Jan 17 06:33:53 pmox kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80
Jan 17 06:33:53 pmox kernel: RIP: 0033:0x7f759a82b960
Jan 17 06:33:53 pmox kernel: RSP: 002b:00007fffd8ad4838 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
Jan 17 06:33:53 pmox kernel: RAX: ffffffffffffffda RBX: 00005f7e60337f50 RCX: 00007f759a82b960
Jan 17 06:33:53 pmox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
Jan 17 06:33:53 pmox kernel: RBP: 00005f7e60337f80 R08: 00007f759a92a160 R09: 0000000000000073
Jan 17 06:33:53 pmox kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
Jan 17 06:33:53 pmox kernel: R13: 00007fffd8ad4a30 R14: 00005f7e5a3f3d68 R15: 00007f759a986020
Jan 17 06:33:53 pmox kernel: </TASK>