Hello.
With the new Proxmox VE 9.1. with the next kernel:
6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z)
We have a constant process stack (stack traces like):
...
INFO: task iou-wrk-730901:813300 <reader> blocked on an rw-semaphore likely owned by task khugepaged:225 <writer>
task:khugepaged state:R running task stack:0 pid:225 tgid:225 ppid:2 task_flags:0x200040 flags:0x00004000
Call Trace:
<TASK>
__schedule+0x468/0x1310
__cond_resched+0x5c/0x80
__cond_resched_rwlock_write+0x4c/0x80
tdp_mmu_zap_leafs+0x1b1/0x240 [kvm]
kvm_tdp_mmu_unmap_gfn_range+0xa8/0xf0 [kvm]
kvm_unmap_gfn_range+0xe0/0x120 [kvm]
kvm_mmu_notifier_invalidate_range_start+0x1ce/0x430 [kvm]
__mmu_notifier_invalidate_range_start+0x94/0x1b0
collapse_huge_page+0x1439/0x16f0
? sysvec_apic_timer_interrupt+0x57/0xc0
hpage_collapse_scan_pmd+0x6bb/0x970
khugepaged+0x79a/0xa10
? __pfx_khugepaged+0x10/0x10
kthread+0x10b/0x220
? __pfx_kthread+0x10/0x10
ret_from_fork+0x208/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
INFO: task iou-wrk-730901:813300 blocked for more than 245 seconds.
Tainted: P IO 6.17.2-1-pve #1
Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:iou-wrk-730901 state
stack:0 pid:813300 tgid:730899 ppid:1 task_flags:0x84040dc flags:0x00004000
Call trace:
<TASK>
__schedule+0x468/0x1310
schedule+0x27/0xf0
schedule_preempt_disabled+0x15/0x30
rwsem_down_read_slowpath+0x24e/0x560
? native_load_gs_index+0x3b/0x60
down_read+0x48/0xc0
do_exit+0x1f2/0xa20
io_wq_worker+0x2d6/0x390
? finish_task_switch.isra.0+0x9c/0x340
? __pfx_io_wq_worker+0x10/0x10
ret_from_fork+0x208/0x240
? __pfx_io_wq_worker+0x10/0x10
ret_from_fork_asm+0x1a/0x30
RIP: 0033:0x0
RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 000000000000010f
RAX: 0000000000000000 RBX: 000071440a0b26c0 RCX: 000071440dcac9ee
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 00007143fc000bb0
RBP: 000071440a0adce0 R08: 0000000000000008 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffffff
R13: 00005b1e9c9bdb10 R14: 000071440a0adce0 R15: 0000000000000000
</TASK>
INFO: task iou-wrk-730901:813300 <reader> blocked on an rw-semaphore likely owned by task khugepaged:225 <writer>
task:khugepaged state:R running task stack:0 pid:225 tgid:225 ppid:2 task_flags:0x200040 flags:0x00004000
Call Trace:
<TASK>
__schedule+0x468/0x1310
__cond_resched+0x5c/0x80
__cond_resched_rwlock_write+0x4c/0x80
tdp_mmu_zap_leafs+0x1b1/0x240 [kvm]
kvm_tdp_mmu_unmap_gfn_range+0xa8/0xf0 [kvm]
kvm_unmap_gfn_range+0xe0/0x120 [kvm]
kvm_mmu_notifier_invalidate_range_start+0x1ce/0x430 [kvm]
__mmu_notifier_invalidate_range_start+0x94/0x1b0
collapse_huge_page+0x1439/0x16f0
? sysvec_apic_timer_interrupt+0x57/0xc0
hpage_collapse_scan_pmd+0x6bb/0x970
khugepaged+0x79a/0xa10
? __pfx_khugepaged+0x10/0x10
kthread+0x10b/0x220
? __pfx_kthread+0x10/0x10
ret_from_fork+0x208/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
INFO: task iou-wrk-730901:813300 blocked for more than 491 seconds.
Tainted: P IO 6.17.2-1-pve #1
Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:iou-wrk-730901 state
stack:0 pid:813300 tgid:730899 ppid:1 [/CODE]
<-- Server is using ECC RAM without the errors.
P.S. Server was normally working before the last kernel upgrade and this reboot.
After new reboot the situation remains the same (with the errors like above)
Ticket: https://bugzilla.proxmox.com/show_bug.cgi?id=7052
BR,
Hrvoje
With the new Proxmox VE 9.1. with the next kernel:
6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z)
We have a constant process stack (stack traces like):
Code:
INFO: task iou-wrk-730901:813300 <reader> blocked on an rw-semaphore likely owned by task khugepaged:225 <writer>
task:khugepaged state:R running task stack:0 pid:225 tgid:225 ppid:2 task_flags:0x200040 flags:0x00004000
Call Trace:
<TASK>
__schedule+0x468/0x1310
__cond_resched+0x5c/0x80
__cond_resched_rwlock_write+0x4c/0x80
tdp_mmu_zap_leafs+0x1b1/0x240 [kvm]
kvm_tdp_mmu_unmap_gfn_range+0xa8/0xf0 [kvm]
kvm_unmap_gfn_range+0xe0/0x120 [kvm]
kvm_mmu_notifier_invalidate_range_start+0x1ce/0x430 [kvm]
__mmu_notifier_invalidate_range_start+0x94/0x1b0
collapse_huge_page+0x1439/0x16f0
? sysvec_apic_timer_interrupt+0x57/0xc0
hpage_collapse_scan_pmd+0x6bb/0x970
khugepaged+0x79a/0xa10
? __pfx_khugepaged+0x10/0x10
kthread+0x10b/0x220
? __pfx_kthread+0x10/0x10
ret_from_fork+0x208/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
INFO: task iou-wrk-730901:813300 blocked for more than 245 seconds.
Tainted: P IO 6.17.2-1-pve #1
Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message
...
INFO: task iou-wrk-730901:813300 <reader> blocked on an rw-semaphore likely owned by task khugepaged:225 <writer>
task:khugepaged state:R running task stack:0 pid:225 tgid:225 ppid:2 task_flags:0x200040 flags:0x00004000
Call Trace:
<TASK>
__schedule+0x468/0x1310
__cond_resched+0x5c/0x80
__cond_resched_rwlock_write+0x4c/0x80
tdp_mmu_zap_leafs+0x1b1/0x240 [kvm]
kvm_tdp_mmu_unmap_gfn_range+0xa8/0xf0 [kvm]
kvm_unmap_gfn_range+0xe0/0x120 [kvm]
kvm_mmu_notifier_invalidate_range_start+0x1ce/0x430 [kvm]
__mmu_notifier_invalidate_range_start+0x94/0x1b0
collapse_huge_page+0x1439/0x16f0
? sysvec_apic_timer_interrupt+0x57/0xc0
hpage_collapse_scan_pmd+0x6bb/0x970
khugepaged+0x79a/0xa10
? __pfx_khugepaged+0x10/0x10
kthread+0x10b/0x220
? __pfx_kthread+0x10/0x10
ret_from_fork+0x208/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
INFO: task iou-wrk-730901:813300 blocked for more than 245 seconds.
Tainted: P IO 6.17.2-1-pve #1
Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:iou-wrk-730901 state
Call trace:
<TASK>
__schedule+0x468/0x1310
schedule+0x27/0xf0
schedule_preempt_disabled+0x15/0x30
rwsem_down_read_slowpath+0x24e/0x560
? native_load_gs_index+0x3b/0x60
down_read+0x48/0xc0
do_exit+0x1f2/0xa20
io_wq_worker+0x2d6/0x390
? finish_task_switch.isra.0+0x9c/0x340
? __pfx_io_wq_worker+0x10/0x10
ret_from_fork+0x208/0x240
? __pfx_io_wq_worker+0x10/0x10
ret_from_fork_asm+0x1a/0x30
RIP: 0033:0x0
RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 000000000000010f
RAX: 0000000000000000 RBX: 000071440a0b26c0 RCX: 000071440dcac9ee
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 00007143fc000bb0
RBP: 000071440a0adce0 R08: 0000000000000008 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffffff
R13: 00005b1e9c9bdb10 R14: 000071440a0adce0 R15: 0000000000000000
</TASK>
INFO: task iou-wrk-730901:813300 <reader> blocked on an rw-semaphore likely owned by task khugepaged:225 <writer>
task:khugepaged state:R running task stack:0 pid:225 tgid:225 ppid:2 task_flags:0x200040 flags:0x00004000
Call Trace:
<TASK>
__schedule+0x468/0x1310
__cond_resched+0x5c/0x80
__cond_resched_rwlock_write+0x4c/0x80
tdp_mmu_zap_leafs+0x1b1/0x240 [kvm]
kvm_tdp_mmu_unmap_gfn_range+0xa8/0xf0 [kvm]
kvm_unmap_gfn_range+0xe0/0x120 [kvm]
kvm_mmu_notifier_invalidate_range_start+0x1ce/0x430 [kvm]
__mmu_notifier_invalidate_range_start+0x94/0x1b0
collapse_huge_page+0x1439/0x16f0
? sysvec_apic_timer_interrupt+0x57/0xc0
hpage_collapse_scan_pmd+0x6bb/0x970
khugepaged+0x79a/0xa10
? __pfx_khugepaged+0x10/0x10
kthread+0x10b/0x220
? __pfx_kthread+0x10/0x10
ret_from_fork+0x208/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
INFO: task iou-wrk-730901:813300 blocked for more than 491 seconds.
Tainted: P IO 6.17.2-1-pve #1
Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:iou-wrk-730901 state
<-- Server is using ECC RAM without the errors.
Code:
pveversion
pve-manager/9.1.1/42db4a6cf33dac83 (running kernel: 6.17.2-1-pve)
P.S. Server was normally working before the last kernel upgrade and this reboot.
After new reboot the situation remains the same (with the errors like above)
Ticket: https://bugzilla.proxmox.com/show_bug.cgi?id=7052
BR,
Hrvoje
Last edited: