rcu: INFO: rcu_sched self-detected stall on CPU

[Alva]

Hello,

I rebooted my host (Dell R720) last night and all of my VM's are now extremely slow and displaying the following in their console:

rcu: INFO: rcu_sched self-detected stall on CPU

I've attached a screen capture from one VM as an example.

I recently updated to the the current release from community subscription: Linux pve01 5.15.30-2-pve #1 SMP PVE 5.15.30-3 (Fri, 22 Apr 2022 18:08:27 +0200) x86_64

Any help would be greatly appreciated.

Thanks!

 

[Alva]

Definitely a kernel v5.15 issue. I've reverted to the previous kernel (v5.13) and pinned. Not a proper fix, but at least it keeps stuff running for now:

proxmox-boot-tool kernel pin 5.13.19-6-pve

Source/credit: https://forum.proxmox.com/threads/gpu-passthrough-issues-after-upgrade-to-7-2.109051/post-468934

 

[Bengt Nolin]

I have also gotten this some some machines after the upgrade to PVE 7.2 and kernel 5.15.

[155118.277548] INFO: rcu_sched self-detected stall on CPU
[155118.277575]      0-...: (2 GPs behind) idle=6ef/1/0 softirq=1813405/1813406 fqs=0
[155118.277590]       (t=334307 jiffies g=1395437 c=1395436 q=1)
[155118.277605] rcu_sched kthread starved for 334307 jiffies! g1395437 c1395436 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[155118.277625] rcu_sched       S    0     7      2 0x00000000
[155118.277628]  0000000000000046 ffff957678425800 0000000000000000 ffff95767bffae80
[155118.277630]  ffff95767fb1da00 ffff95767b804180 ffffb954c0667db0 ffffffff8781eeb9
[155118.277631]  ffffb954c0667de0 0000000102497bd5 ffff95767fb1da00 0000000000000001
[155118.277633] Call Trace:
[155118.277638]  [<ffffffff8781eeb9>] ? __schedule+0x239/0x6f0
[155118.277639]  [<ffffffff8781f3a2>] ? schedule+0x32/0x80
[155118.277641]  [<ffffffff878227d7>] ? schedule_timeout+0x167/0x380
[155118.277642]  [<ffffffff872e9b10>] ? del_timer_sync+0x50/0x50
[155118.277643]  [<ffffffff872e30a5>] ? rcu_gp_kthread+0x505/0x850
[155118.277645]  [<ffffffff872bdd6f>] ? __wake_up_common+0x4f/0x90
[155118.277646]  [<ffffffff872e2ba0>] ? get_state_synchronize_rcu+0x10/0x10
[155118.277648]  [<ffffffff8729b119>] ? kthread+0xd9/0xf0
[155118.277649]  [<ffffffff8729b040>] ? kthread_park+0x60/0x60
[155118.277650]  [<ffffffff87823f37>] ? ret_from_fork+0x57/0x70
[155118.277656] Task dump for CPU 0:
[155118.277656] swapper/0       R  running task        0     0      0 0x00000008
[155118.277658]  ffffffff87f19e00 ffffffff872a8afb 0000000000000000 ffffffff87f19e00
[155118.277659]  ffffffff87813a2a ffff95767fa1e740 ffffffff87e4fe00 0000000000000000
[155118.277661]  ffffffff87f19e00 00000000ffffffff ffffffff872e4a1b ffff95767fa1da00
[155118.277662] Call Trace:
[155118.277662]  <IRQ>
[155118.277664]  [<ffffffff872a8afb>] ? sched_show_task+0xcb/0x130
[155118.277665]  [<ffffffff87813a2a>] ? rcu_dump_cpu_stacks+0x92/0xb2
[155118.277666]  [<ffffffff872e4a1b>] ? rcu_check_callbacks+0x77b/0x8d0
[155118.277667]  [<ffffffff872a562e>] ? check_preempt_curr+0x4e/0x90
[155118.277669]  [<ffffffff872ac4fd>] ? account_process_tick+0xbd/0x130
[155118.277670]  [<ffffffff872fb070>] ? tick_sched_do_timer+0x30/0x30
[155118.277671]  [<ffffffff872eb648>] ? update_process_times+0x28/0x50
[155118.277672]  [<ffffffff872faa70>] ? tick_sched_handle.isra.12+0x20/0x50
[155118.277673]  [<ffffffff872fb0a8>] ? tick_sched_timer+0x38/0x70
[155118.277674]  [<ffffffff872ec11e>] ? __hrtimer_run_queues+0xde/0x250
[155118.277675]  [<ffffffff872ec7fc>] ? hrtimer_interrupt+0x9c/0x1a0
[155118.277676]  [<ffffffff878274e7>] ? smp_apic_timer_interrupt+0x47/0x60
[155118.277677]  [<ffffffff87825c1e>] ? apic_timer_interrupt+0x9e/0xb0
[155118.277678]  <EOI>
[155118.277679]  [<ffffffff87823300>] ? __cpuidle_text_start+0x8/0x8
[155118.277680]  [<ffffffff878235ee>] ? native_safe_halt+0xe/0x10
[155118.277681]  [<ffffffff8782331a>] ? default_idle+0x1a/0xd0
[155118.277682]  [<ffffffff872beb8a>] ? cpu_startup_entry+0x1ca/0x240
[155118.277684]  [<ffffffff87f43f10>] ? start_kernel+0x44f/0x472
[155118.277685]  [<ffffffff87f43120>] ? early_idt_handler_array+0x120/0x120
[155118.277686]  [<ffffffff87f433a9>] ? x86_64_start_kernel+0x14c/0x170