rcu: INFO: rcu_sched self-detected stall on CPU

Alva

Member
Mar 31, 2022
15
3
8
Hello,

I rebooted my host (Dell R720) last night and all of my VM's are now extremely slow and displaying the following in their console:

Code:
rcu: INFO: rcu_sched self-detected stall on CPU

I've attached a screen capture from one VM as an example.

I recently updated to the the current release from community subscription: Linux pve01 5.15.30-2-pve #1 SMP PVE 5.15.30-3 (Fri, 22 Apr 2022 18:08:27 +0200) x86_64

Any help would be greatly appreciated.

Thanks!
 

Attachments

  • Console.png
    Console.png
    276.5 KB · Views: 53
Last edited:
I have also gotten this some some machines after the upgrade to PVE 7.2 and kernel 5.15.

Code:
[155118.277548] INFO: rcu_sched self-detected stall on CPU
[155118.277575]      0-...: (2 GPs behind) idle=6ef/1/0 softirq=1813405/1813406 fqs=0
[155118.277590]       (t=334307 jiffies g=1395437 c=1395436 q=1)
[155118.277605] rcu_sched kthread starved for 334307 jiffies! g1395437 c1395436 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[155118.277625] rcu_sched       S    0     7      2 0x00000000
[155118.277628]  0000000000000046 ffff957678425800 0000000000000000 ffff95767bffae80
[155118.277630]  ffff95767fb1da00 ffff95767b804180 ffffb954c0667db0 ffffffff8781eeb9
[155118.277631]  ffffb954c0667de0 0000000102497bd5 ffff95767fb1da00 0000000000000001
[155118.277633] Call Trace:
[155118.277638]  [<ffffffff8781eeb9>] ? __schedule+0x239/0x6f0
[155118.277639]  [<ffffffff8781f3a2>] ? schedule+0x32/0x80
[155118.277641]  [<ffffffff878227d7>] ? schedule_timeout+0x167/0x380
[155118.277642]  [<ffffffff872e9b10>] ? del_timer_sync+0x50/0x50
[155118.277643]  [<ffffffff872e30a5>] ? rcu_gp_kthread+0x505/0x850
[155118.277645]  [<ffffffff872bdd6f>] ? __wake_up_common+0x4f/0x90
[155118.277646]  [<ffffffff872e2ba0>] ? get_state_synchronize_rcu+0x10/0x10
[155118.277648]  [<ffffffff8729b119>] ? kthread+0xd9/0xf0
[155118.277649]  [<ffffffff8729b040>] ? kthread_park+0x60/0x60
[155118.277650]  [<ffffffff87823f37>] ? ret_from_fork+0x57/0x70
[155118.277656] Task dump for CPU 0:
[155118.277656] swapper/0       R  running task        0     0      0 0x00000008
[155118.277658]  ffffffff87f19e00 ffffffff872a8afb 0000000000000000 ffffffff87f19e00
[155118.277659]  ffffffff87813a2a ffff95767fa1e740 ffffffff87e4fe00 0000000000000000
[155118.277661]  ffffffff87f19e00 00000000ffffffff ffffffff872e4a1b ffff95767fa1da00
[155118.277662] Call Trace:
[155118.277662]  <IRQ>
[155118.277664]  [<ffffffff872a8afb>] ? sched_show_task+0xcb/0x130
[155118.277665]  [<ffffffff87813a2a>] ? rcu_dump_cpu_stacks+0x92/0xb2
[155118.277666]  [<ffffffff872e4a1b>] ? rcu_check_callbacks+0x77b/0x8d0
[155118.277667]  [<ffffffff872a562e>] ? check_preempt_curr+0x4e/0x90
[155118.277669]  [<ffffffff872ac4fd>] ? account_process_tick+0xbd/0x130
[155118.277670]  [<ffffffff872fb070>] ? tick_sched_do_timer+0x30/0x30
[155118.277671]  [<ffffffff872eb648>] ? update_process_times+0x28/0x50
[155118.277672]  [<ffffffff872faa70>] ? tick_sched_handle.isra.12+0x20/0x50
[155118.277673]  [<ffffffff872fb0a8>] ? tick_sched_timer+0x38/0x70
[155118.277674]  [<ffffffff872ec11e>] ? __hrtimer_run_queues+0xde/0x250
[155118.277675]  [<ffffffff872ec7fc>] ? hrtimer_interrupt+0x9c/0x1a0
[155118.277676]  [<ffffffff878274e7>] ? smp_apic_timer_interrupt+0x47/0x60
[155118.277677]  [<ffffffff87825c1e>] ? apic_timer_interrupt+0x9e/0xb0
[155118.277678]  <EOI>
[155118.277679]  [<ffffffff87823300>] ? __cpuidle_text_start+0x8/0x8
[155118.277680]  [<ffffffff878235ee>] ? native_safe_halt+0xe/0x10
[155118.277681]  [<ffffffff8782331a>] ? default_idle+0x1a/0xd0
[155118.277682]  [<ffffffff872beb8a>] ? cpu_startup_entry+0x1ca/0x240
[155118.277684]  [<ffffffff87f43f10>] ? start_kernel+0x44f/0x472
[155118.277685]  [<ffffffff87f43120>] ? early_idt_handler_array+0x120/0x120
[155118.277686]  [<ffffffff87f433a9>] ? x86_64_start_kernel+0x14c/0x170
 
  • Like
Reactions: zeuxprox

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!