Proxmox friert sporadisch einfach ein

[hansanahb]

Moin zusammen,
ich habe seit mehreren Monaten das Problem, dass mein Proxmox-Server spontan einfach einfriert. Manchmal läuft er ein paar Tage, manchmal nur ein paar Stunden. Ich habe die ganze Kiste schon platt gemacht, neu aufgesetzt, diverse Hardware auf Verdacht durchgetauscht, komme aber aktuell nicht weiter.

Eckdaten des Systems:

Mainboard	Gigabyte AX370-Gaming K3 mit BIOS F52
RAM	64Gb 3200Mhz
Festplatten	1x NVME 1Tb, 1x Crucial SSD 2Tb
CPU	Ryzen 7 1700X
GPU	RX 550 Gaming OC
Netzteil	Be Quiet 500W
Proxmox-Version	8.2.2

Wenn ich einen Monitor anschließe tauchen Fehlermeldungen nach folgendem Schema auf (sobald diese auftauchen geht nichts mehr, muss ich komplett neu starten dann die Kiste:

nmi_backtrace_stall_check: CPU 2: NMIs are not reaching exc_nmi() handler, last activity: 21343242424 jiffies ago.
watchdog: BUG: soft lockup - CPU#13 stuck for 23s [kworker/13:2:9313]
usw.

im journal finden sich außerdem folgende logs:

Jul 21 11:59:40 qsfdbs kernel: Sending NMI from CPU 12 to CPUs 14:
Jul 21 11:59:40 qsfdbs kernel: watchdog: BUG: soft lockup - CPU#13 stuck for 49s! [kworker/13:2:9313]
Jul 21 11:59:40 qsfdbs kernel: Modules linked in: ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter nf_tables bonding tls softdog sunrpc nfnetlink_log binfmt_misc nfnetlink intel_rapl_msr intel_rapl_commo>
Jul 21 11:59:40 qsfdbs kernel: mdio libahci nvme_auth wmi gpio_amdpt [last unloaded: cpuid]
Jul 21 11:59:40 qsfdbs kernel: CPU: 13 PID: 9313 Comm: kworker/13:2 Tainted: P O L 6.8.4-2-pve #1
Jul 21 11:59:40 qsfdbs kernel: Hardware name: Gigabyte Technology Co., Ltd. AX370-Gaming K3/AX370-Gaming K3, BIOS F52 03/22/2024
Jul 21 11:59:40 qsfdbs kernel: Workqueue: events netstamp_clear
Jul 21 11:59:40 qsfdbs kernel: RIP: 0010:smp_call_function_many_cond+0x136/0x500
Jul 21 11:59:40 qsfdbs kernel: Code: 63 d0 e8 3d 3e 5d 00 3b 05 37 9b 38 02 73 25 48 63 d0 49 8b 37 48 03 34 d5 e0 ac 2a a5 8b 56 08 83 e2 01 74 0a f3 90 8b 4e 08 <83> e1 01 75 f6 83 c0 01 eb c1 48 83 c4 48 5b 41 5c 41 5d 41 5e 41
Jul 21 11:59:40 qsfdbs kernel: RSP: 0018:ffffabb541b4bca0 EFLAGS: 00000202
Jul 21 11:59:40 qsfdbs kernel: RAX: 0000000000000002 RBX: 0000000000000246 RCX: 0000000000000011
Jul 21 11:59:40 qsfdbs kernel: RDX: 0000000000000001 RSI: ffff8c4f7e33ddc0 RDI: 0000000000000000
Jul 21 11:59:40 qsfdbs kernel: RBP: ffffabb541b4bd10 R08: 0000000000000000 R09: 0000000000000000
Jul 21 11:59:40 qsfdbs kernel: R10: ffff8c4080906158 R11: 0000000000000000 R12: ffff8c4f7e8b5e80
Jul 21 11:59:40 qsfdbs kernel: R13: 0000000000000001 R14: 000000000000000d R15: ffff8c4f7e8b5e80
Jul 21 11:59:40 qsfdbs kernel: FS: 0000000000000000(0000) GS:ffff8c4f7e880000(0000) knlGS:0000000000000000
Jul 21 11:59:40 qsfdbs kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 21 11:59:40 qsfdbs kernel: CR2: 00006498a3a0f7b0 CR3: 0000000face36000 CR4: 00000000003506f0
Jul 21 11:59:40 qsfdbs kernel: Call Trace:
Jul 21 11:59:40 qsfdbs kernel: <IRQ>
Jul 21 11:59:40 qsfdbs kernel: ? show_regs+0x6d/0x80
Jul 21 11:59:40 qsfdbs kernel: ? watchdog_timer_fn+0x206/0x290
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_watchdog_timer_fn+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: ? __hrtimer_run_queues+0x108/0x280
Jul 21 11:59:40 qsfdbs kernel: ? srso_return_thunk+0x5/0x5f
Jul 21 11:59:40 qsfdbs kernel: ? hrtimer_interrupt+0xf6/0x250
Jul 21 11:59:40 qsfdbs kernel: ? __sysvec_apic_timer_interrupt+0x51/0x150
Jul 21 11:59:12 qsfdbs kernel: static_key_enable+0x1a/0x30
Jul 21 11:59:12 qsfdbs kernel: netstamp_clear+0x2d/0x50
Jul 21 11:59:12 qsfdbs kernel: process_one_work+0x16d/0x350
Jul 21 11:59:12 qsfdbs kernel: worker_thread+0x306/0x440
Jul 21 11:59:12 qsfdbs kernel: ? __pfx_worker_thread+0x10/0x10
Jul 21 11:59:12 qsfdbs kernel: kthread+0xf2/0x120
Jul 21 11:59:12 qsfdbs kernel: ? __pfx_kthread+0x10/0x10
Jul 21 11:59:12 qsfdbs kernel: ret_from_fork+0x47/0x70
Jul 21 11:59:12 qsfdbs kernel: ? __pfx_kthread+0x10/0x10
Jul 21 11:59:12 qsfdbs kernel: ret_from_fork_asm+0x1b/0x30
Jul 21 11:59:12 qsfdbs kernel: </TASK>
Jul 21 11:59:40 qsfdbs kernel: nmi_backtrace_stall_check: CPU 7: NMIs are not reaching exc_nmi() handler, last activity: 101334 jiffies ago.
Jul 21 11:59:40 qsfdbs kernel: Sending NMI from CPU 12 to CPUs 10:
Jul 21 11:59:40 qsfdbs kernel: nmi_backtrace_stall_check: CPU 10: NMIs are not reaching exc_nmi() handler, last activity: 4298117972 jiffies ago.
Jul 21 11:59:40 qsfdbs kernel: Sending NMI from CPU 12 to CPUs 15:
Jul 21 11:59:40 qsfdbs kernel: nmi_backtrace_stall_check: CPU 15: NMIs are not reaching exc_nmi() handler, last activity: 559910 jiffies ago.
Jul 21 11:59:40 qsfdbs kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 110008 jiffies! g93877 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
Jul 21 11:59:40 qsfdbs kernel: rcu: Possible timer handling issue on cpu=14 timer-softirq=11673
Jul 21 11:59:40 qsfdbs kernel: rcu: rcu_preempt kthread starved for 110017 jiffies! g93877 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=14
Jul 21 11:59:40 qsfdbs kernel: rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Jul 21 11:59:40 qsfdbs kernel: rcu: RCU grace-period kthread stack dump:
Jul 21 11:59:40 qsfdbs kernel: task:rcu_preempt state:I stack:0 pid:17 tgid:17 ppid:2 flags:0x00004000
Jul 21 11:59:40 qsfdbs kernel: Call Trace:
Jul 21 11:59:40 qsfdbs kernel: <TASK>
Jul 21 11:59:40 qsfdbs kernel: __schedule+0x401/0x15e0
Jul 21 11:59:40 qsfdbs kernel: ? srso_return_thunk+0x5/0x5f
Jul 21 11:59:40 qsfdbs kernel: schedule+0x33/0x110
Jul 21 11:59:40 qsfdbs kernel: schedule_timeout+0x95/0x170
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_process_timeout+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: rcu_gp_fqs_loop+0x13d/0x550
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: rcu_gp_kthread+0xd3/0x1a0
Jul 21 11:59:40 qsfdbs kernel: kthread+0xf2/0x120
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_kthread+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: ret_from_fork+0x47/0x70
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_kthread+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: ret_from_fork_asm+0x1b/0x30
Jul 21 11:59:40 qsfdbs kernel: </TASK>
Jul 21 11:59:40 qsfdbs kernel: rcu: Stack dump where RCU GP kthread last ran:
Jul 21 11:59:40 qsfdbs kernel: Sending NMI from CPU 12 to CPUs 14:
Jul 21 11:59:40 qsfdbs kernel: watchdog: BUG: soft lockup - CPU#13 stuck for 49s! [kworker/13:2:9313]
Jul 21 11:59:40 qsfdbs kernel: Modules linked in: ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter nf_tables bonding tls softdog sunrpc nfnetlink_log binfmt_misc nfnetlink intel_rapl_msr intel_rapl_commo>
Jul 21 11:59:40 qsfdbs kernel: mdio libahci nvme_auth wmi gpio_amdpt [last unloaded: cpuid]
Jul 21 11:59:40 qsfdbs kernel: CPU: 13 PID: 9313 Comm: kworker/13:2 Tainted: P O L 6.8.4-2-pve #1
Jul 21 11:59:40 qsfdbs kernel: Hardware name: Gigabyte Technology Co., Ltd. AX370-Gaming K3/AX370-Gaming K3, BIOS F52 03/22/2024
Jul 21 11:59:40 qsfdbs kernel: Workqueue: events netstamp_clear
Jul 21 11:59:40 qsfdbs kernel: RIP: 0010:smp_call_function_many_cond+0x136/0x500
Jul 21 11:59:40 qsfdbs kernel: Code: 63 d0 e8 3d 3e 5d 00 3b 05 37 9b 38 02 73 25 48 63 d0 49 8b 37 48 03 34 d5 e0 ac 2a a5 8b 56 08 83 e2 01 74 0a f3 90 8b 4e 08 <83> e1 01 75 f6 83 c0 01 eb c1 48 83 c4 48 5b 41 5c 41 5d 41 5e 41
Jul 21 11:59:40 qsfdbs kernel: RSP: 0018:ffffabb541b4bca0 EFLAGS: 00000202
Jul 21 11:59:40 qsfdbs kernel: RAX: 0000000000000002 RBX: 0000000000000246 RCX: 0000000000000011
Jul 21 11:59:40 qsfdbs kernel: RDX: 0000000000000001 RSI: ffff8c4f7e33ddc0 RDI: 0000000000000000
Jul 21 11:59:40 qsfdbs kernel: RBP: ffffabb541b4bd10 R08: 0000000000000000 R09: 0000000000000000
Jul 21 11:59:40 qsfdbs kernel: R10: ffff8c4080906158 R11: 0000000000000000 R12: ffff8c4f7e8b5e80
Jul 21 11:59:40 qsfdbs kernel: R13: 0000000000000001 R14: 000000000000000d R15: ffff8c4f7e8b5e80
Jul 21 11:59:40 qsfdbs kernel: FS: 0000000000000000(0000) GS:ffff8c4f7e880000(0000) knlGS:0000000000000000
Jul 21 11:59:40 qsfdbs kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 21 11:59:40 qsfdbs kernel: CR2: 00006498a3a0f7b0 CR3: 0000000face36000 CR4: 00000000003506f0
Jul 21 11:59:40 qsfdbs kernel: Call Trace:
Jul 21 11:59:40 qsfdbs kernel: <IRQ>
Jul 21 11:59:40 qsfdbs kernel: ? show_regs+0x6d/0x80
Jul 21 11:59:40 qsfdbs kernel: ? watchdog_timer_fn+0x206/0x290
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_watchdog_timer_fn+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: ? __hrtimer_run_queues+0x108/0x280
Jul 21 11:59:40 qsfdbs kernel: ? srso_return_thunk+0x5/0x5f
Jul 21 11:59:40 qsfdbs kernel: ? hrtimer_interrupt+0xf6/0x250
Jul 21 11:59:40 qsfdbs kernel: ? __sysvec_apic_timer_interrupt+0x51/0x150
Jul 21 11:59:40 qsfdbs kernel: ? sysvec_apic_timer_interrupt+0x8d/0xd0
Jul 21 11:59:40 qsfdbs kernel: </IRQ>
Jul 21 11:59:40 qsfdbs kernel: <TASK>
Jul 21 11:59:40 qsfdbs kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jul 21 11:59:40 qsfdbs kernel: ? smp_call_function_many_cond+0x136/0x500
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_do_sync_core+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: on_each_cpu_cond_mask+0x24/0x60
Jul 21 11:59:40 qsfdbs kernel: text_poke_bp_batch+0xbe/0x300
Jul 21 11:59:40 qsfdbs kernel: text_poke_finish+0x1f/0x40
Jul 21 11:59:40 qsfdbs kernel: arch_jump_label_transform_apply+0x1a/0x30
Jul 21 11:59:40 qsfdbs kernel: __jump_label_update+0xf4/0x140
Jul 21 11:59:40 qsfdbs kernel: jump_label_update+0xae/0x120
Jul 21 11:59:40 qsfdbs kernel: static_key_enable_cpuslocked+0x87/0xb0
Jul 21 11:59:40 qsfdbs kernel: static_key_enable+0x1a/0x30
Jul 21 11:59:40 qsfdbs kernel: netstamp_clear+0x2d/0x50
Jul 21 11:59:40 qsfdbs kernel: process_one_work+0x16d/0x350
Jul 21 11:59:40 qsfdbs kernel: worker_thread+0x306/0x440
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_worker_thread+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: kthread+0xf2/0x120
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_kthread+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: ret_from_fork+0x47/0x70
Jul 21 11:59:40 qsfdbs kernel: ? __pfx_kthread+0x10/0x10
Jul 21 11:59:40 qsfdbs kernel: ret_from_fork_asm+0x1b/0x30
Jul 21 11:59:40 qsfdbs kernel: </TASK>
Jul 21 11:59:43 qsfdbs kernel: nmi_backtrace_stall_check: CPU 14: NMIs are not reaching exc_nmi() handler, last activity: 4298137977 jiffies ago.
Jul 21 12:00:08 qsfdbs kernel: watchdog: BUG: soft lockup - CPU#13 stuck for 75s! [kworker/13:2:9313]
Jul 21 12:00:08 qsfdbs kernel: Modules linked in: ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter nf_tables bonding tls softdog sunrpc nfnetlink_log binfmt_misc nfnetlink intel_rapl_msr intel_rapl_commo>
Jul 21 12:00:08 qsfdbs kernel: mdio libahci nvme_auth wmi gpio_amdpt [last unloaded: cpuid]
Jul 21 12:00:08 qsfdbs kernel: CPU: 13 PID: 9313 Comm: kworker/13:2 Tainted: P O L 6.8.4-2-pve #1
Jul 21 12:00:08 qsfdbs kernel: Hardware name: Gigabyte Technology Co., Ltd. AX370-Gaming K3/AX370-Gaming K3, BIOS F52 03/22/2024
Jul 21 12:00:08 qsfdbs kernel: Workqueue: events netstamp_clear
Jul 21 12:00:08 qsfdbs kernel: RIP: 0010:smp_call_function_many_cond+0x133/0x500
Jul 21 12:00:08 qsfdbs kernel: Code: 7f 08 48 63 d0 e8 3d 3e 5d 00 3b 05 37 9b 38 02 73 25 48 63 d0 49 8b 37 48 03 34 d5 e0 ac 2a a5 8b 56 08 83 e2 01 74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb c1 48 83 c4 48 5b 41 5c 41 5d
Jul 21 12:00:08 qsfdbs kernel: RSP: 0018:ffffabb541b4bca0 EFLAGS: 00000202
Jul 21 12:00:08 qsfdbs kernel: RAX: 0000000000000002 RBX: 0000000000000246 RCX: 0000000000000001
Jul 21 12:00:08 qsfdbs kernel: RDX: 0000000000000001 RSI: ffff8c4f7e33ddc0 RDI: 0000000000000000
Jul 21 12:00:08 qsfdbs kernel: RBP: ffffabb541b4bd10 R08: 0000000000000000 R09: 0000000000000000
Jul 21 12:00:08 qsfdbs kernel: R10: ffff8c4080906158 R11: 0000000000000000 R12: ffff8c4f7e8b5e80
Jul 21 12:00:08 qsfdbs kernel: R13: 0000000000000001 R14: 000000000000000d R15: ffff8c4f7e8b5e80
Jul 21 12:00:08 qsfdbs kernel: FS: 0000000000000000(0000) GS:ffff8c4f7e880000(0000) knlGS:0000000000000000
Jul 21 12:00:08 qsfdbs kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 21 12:00:08 qsfdbs kernel: CR2: 00006498a3a0f7b0 CR3: 0000000face36000 CR4: 00000000003506f0
Jul 21 12:00:08 qsfdbs kernel: Call Trace:
Jul 21 12:00:08 qsfdbs kernel: <IRQ>
Jul 21 12:00:08 qsfdbs kernel: ? show_regs+0x6d/0x80
Jul 21 12:00:08 qsfdbs kernel: ? watchdog_timer_fn+0x206/0x290
Jul 21 12:00:08 qsfdbs kernel: ? __pfx_watchdog_timer_fn+0x10/0x10
Jul 21 12:00:08 qsfdbs kernel: ? __hrtimer_run_queues+0x108/0x280
Jul 21 12:00:08 qsfdbs kernel: ? srso_return_thunk+0x5/0x5f
Jul 21 12:00:08 qsfdbs kernel: ? hrtimer_interrupt+0xf6/0x250
Jul 21 12:00:08 qsfdbs kernel: ? __sysvec_apic_timer_interrupt+0x51/0x150
Jul 21 12:00:08 qsfdbs kernel: ? sysvec_apic_timer_interrupt+0x8d/0xd0
Jul 21 12:00:08 qsfdbs kernel: </IRQ>
Jul 21 12:00:08 qsfdbs kernel: <TASK>
Jul 21 12:00:08 qsfdbs kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jul 21 12:00:08 qsfdbs kernel: ? smp_call_function_many_cond+0x133/0x500
Jul 21 12:00:08 qsfdbs kernel: ? __pfx_do_sync_core+0x10/0x10
Jul 21 12:00:08 qsfdbs kernel: on_each_cpu_cond_mask+0x24/0x60
Jul 21 12:00:08 qsfdbs kernel: text_poke_bp_batch+0xbe/0x300
Jul 21 12:00:08 qsfdbs kernel: text_poke_finish+0x1f/0x40
Jul 21 12:00:08 qsfdbs kernel: arch_jump_label_transform_apply+0x1a/0x30
Jul 21 12:00:08 qsfdbs kernel: __jump_label_update+0xf4/0x140
Jul 21 12:00:08 qsfdbs kernel: jump_label_update+0xae/0x120
Jul 21 12:00:08 qsfdbs kernel: static_key_enable_cpuslocked+0x87/0xb0
Jul 21 12:00:08 qsfdbs kernel: static_key_enable+0x1a/0x30
Jul 21 12:00:08 qsfdbs kernel: netstamp_clear+0x2d/0x50

Vielleicht hat ja von euch noch jemand eine Idee, wie ich das weiter debuggen und dem Problem auf die Spur kommen kann.

 

[Max Carrara]

Hast du deine CPU auch schon mal getauscht? Ich hatte mal ein ähnliches Problem mit einem Ryzen 7 1700 - in meinem Fall hat sich herausgestellt, dass die CPU einen Hardware-Defekt hatte.

Hast du auch schon mal einen memtest gemacht?