This morning one of our Windows VM froze during a Windows Update and didn't respond anymore. I set the state in HA from started to stopped and stopped the VM manually. Next I set the HA state from stopped to started. The VM tried to start but failed. Proxmox then migrated the VM to another host were it started without any issues. I stopped the Windows Update from installing again. All of the other VMs were not affected on the host. The host were the VM crashed, has now incresed IO delay. Around 6:00 am it was at 0.15% IO delay. At 7:30am it is around 8.3%.
Here is the log from the host this morning. The second part of the log is in the first reply of this post.
Here is the log from the host this morning. The second part of the log is in the first reply of this post.
Code:
Feb 24 06:03:24 pve1 systemd[1]: Starting Daily apt upgrade and clean activities...
Feb 24 06:03:25 pve1 systemd[1]: apt-daily-upgrade.service: Succeeded.
Feb 24 06:03:25 pve1 systemd[1]: Finished Daily apt upgrade and clean activities.
Feb 24 06:17:01 pve1 CRON[3213731]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 24 06:17:01 pve1 CRON[3213732]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 24 06:17:01 pve1 CRON[3213731]: pam_unix(cron:session): session closed for user root
Feb 24 06:24:19 pve1 kernel: INFO: task worker:3212776 blocked for more than 120 seconds.
Feb 24 06:24:19 pve1 kernel: Tainted: P O 5.15.85-1-pve #1
Feb 24 06:24:19 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 24 06:24:19 pve1 kernel: task:worker state:D stack: 0 pid:3212776 ppid: 1 flags:0x00004000
Feb 24 06:24:19 pve1 kernel: Call Trace:
Feb 24 06:24:19 pve1 kernel: <TASK>
Feb 24 06:24:19 pve1 kernel: __schedule+0x34e/0x1740
Feb 24 06:24:19 pve1 kernel: ? release_pages+0x162/0x530
Feb 24 06:24:19 pve1 kernel: schedule+0x69/0x110
Feb 24 06:24:19 pve1 kernel: io_schedule+0x46/0x80
Feb 24 06:24:19 pve1 kernel: wait_on_page_bit_common+0x114/0x3e0
Feb 24 06:24:19 pve1 kernel: ? filemap_invalidate_unlock_two+0x50/0x50
Feb 24 06:24:19 pve1 kernel: wait_on_page_bit+0x3f/0x50
Feb 24 06:24:19 pve1 kernel: wait_on_page_writeback+0x26/0x80
Feb 24 06:24:19 pve1 kernel: __filemap_fdatawait_range+0x97/0x120
Feb 24 06:24:19 pve1 kernel: file_write_and_wait_range+0xd3/0xf0
Feb 24 06:24:19 pve1 kernel: cifs_strict_fsync+0x3b/0x430 [cifs]
Feb 24 06:24:19 pve1 kernel: __x64_sys_fdatasync+0x4c/0x90
Feb 24 06:24:19 pve1 kernel: do_syscall_64+0x5c/0xc0
Feb 24 06:24:19 pve1 kernel: ? syscall_exit_to_user_mode+0x27/0x50
Feb 24 06:24:19 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:24:19 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:24:19 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:24:19 pve1 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Feb 24 06:24:19 pve1 kernel: RIP: 0033:0x7f108289e4ab
Feb 24 06:24:19 pve1 kernel: RSP: 002b:00007f0cf6cf5330 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Feb 24 06:24:19 pve1 kernel: RAX: ffffffffffffffda RBX: 000055bed9479070 RCX: 00007f108289e4ab
Feb 24 06:24:19 pve1 kernel: RDX: 0000000000000001 RSI: 0000000000000081 RDI: 0000000000000026
Feb 24 06:24:19 pve1 kernel: RBP: 00007f1068003500 R08: 0000000000000000 R09: 00000000ffffffff
Feb 24 06:24:19 pve1 kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
Feb 24 06:24:19 pve1 kernel: R13: 00007f1068003578 R14: 000055bed704cb05 R15: 0000000000802000
Feb 24 06:24:19 pve1 kernel: </TASK>
Feb 24 06:25:01 pve1 CRON[3229538]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 24 06:25:01 pve1 CRON[3229539]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Feb 24 06:25:01 pve1 CRON[3229538]: pam_unix(cron:session): session closed for user root
Feb 24 06:26:20 pve1 kernel: INFO: task worker:3212776 blocked for more than 241 seconds.
Feb 24 06:26:20 pve1 kernel: Tainted: P O 5.15.85-1-pve #1
Feb 24 06:26:20 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 24 06:26:20 pve1 kernel: task:worker state:D stack: 0 pid:3212776 ppid: 1 flags:0x00004000
Feb 24 06:26:20 pve1 kernel: Call Trace:
Feb 24 06:26:20 pve1 kernel: <TASK>
Feb 24 06:26:20 pve1 kernel: __schedule+0x34e/0x1740
Feb 24 06:26:20 pve1 kernel: ? release_pages+0x162/0x530
Feb 24 06:26:20 pve1 kernel: schedule+0x69/0x110
Feb 24 06:26:20 pve1 kernel: io_schedule+0x46/0x80
Feb 24 06:26:20 pve1 kernel: wait_on_page_bit_common+0x114/0x3e0
Feb 24 06:26:20 pve1 kernel: ? filemap_invalidate_unlock_two+0x50/0x50
Feb 24 06:26:20 pve1 kernel: wait_on_page_bit+0x3f/0x50
Feb 24 06:26:20 pve1 kernel: wait_on_page_writeback+0x26/0x80
Feb 24 06:26:20 pve1 kernel: __filemap_fdatawait_range+0x97/0x120
Feb 24 06:26:20 pve1 kernel: file_write_and_wait_range+0xd3/0xf0
Feb 24 06:26:20 pve1 kernel: cifs_strict_fsync+0x3b/0x430 [cifs]
Feb 24 06:26:20 pve1 kernel: __x64_sys_fdatasync+0x4c/0x90
Feb 24 06:26:20 pve1 kernel: do_syscall_64+0x5c/0xc0
Feb 24 06:26:20 pve1 kernel: ? syscall_exit_to_user_mode+0x27/0x50
Feb 24 06:26:20 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:26:20 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:26:20 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:26:20 pve1 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Feb 24 06:26:20 pve1 kernel: RIP: 0033:0x7f108289e4ab
Feb 24 06:26:20 pve1 kernel: RSP: 002b:00007f0cf6cf5330 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Feb 24 06:26:20 pve1 kernel: RAX: ffffffffffffffda RBX: 000055bed9479070 RCX: 00007f108289e4ab
Feb 24 06:26:20 pve1 kernel: RDX: 0000000000000001 RSI: 0000000000000081 RDI: 0000000000000026
Feb 24 06:26:20 pve1 kernel: RBP: 00007f1068003500 R08: 0000000000000000 R09: 00000000ffffffff
Feb 24 06:26:20 pve1 kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
Feb 24 06:26:20 pve1 kernel: R13: 00007f1068003578 R14: 000055bed704cb05 R15: 0000000000802000
Feb 24 06:26:20 pve1 kernel: </TASK>
Feb 24 06:28:20 pve1 kernel: INFO: task worker:3212776 blocked for more than 362 seconds.
Feb 24 06:28:20 pve1 kernel: Tainted: P O 5.15.85-1-pve #1
Feb 24 06:28:20 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 24 06:28:20 pve1 kernel: task:worker state:D stack: 0 pid:3212776 ppid: 1 flags:0x00004000
Feb 24 06:28:20 pve1 kernel: Call Trace:
Feb 24 06:28:20 pve1 kernel: <TASK>
Feb 24 06:28:20 pve1 kernel: __schedule+0x34e/0x1740
Feb 24 06:28:20 pve1 kernel: ? release_pages+0x162/0x530
Feb 24 06:28:20 pve1 kernel: schedule+0x69/0x110
Feb 24 06:28:20 pve1 kernel: io_schedule+0x46/0x80
Feb 24 06:28:20 pve1 kernel: wait_on_page_bit_common+0x114/0x3e0
Feb 24 06:28:20 pve1 kernel: ? filemap_invalidate_unlock_two+0x50/0x50
Feb 24 06:28:20 pve1 kernel: wait_on_page_bit+0x3f/0x50
Feb 24 06:28:20 pve1 kernel: wait_on_page_writeback+0x26/0x80
Feb 24 06:28:20 pve1 kernel: __filemap_fdatawait_range+0x97/0x120
Feb 24 06:28:20 pve1 kernel: file_write_and_wait_range+0xd3/0xf0
Feb 24 06:28:20 pve1 kernel: cifs_strict_fsync+0x3b/0x430 [cifs]
Feb 24 06:28:20 pve1 kernel: __x64_sys_fdatasync+0x4c/0x90
Feb 24 06:28:20 pve1 kernel: do_syscall_64+0x5c/0xc0
Feb 24 06:28:20 pve1 kernel: ? syscall_exit_to_user_mode+0x27/0x50
Feb 24 06:28:20 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:28:20 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:28:20 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:28:20 pve1 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Feb 24 06:28:20 pve1 kernel: RIP: 0033:0x7f108289e4ab
Feb 24 06:28:20 pve1 kernel: RSP: 002b:00007f0cf6cf5330 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Feb 24 06:28:20 pve1 kernel: RAX: ffffffffffffffda RBX: 000055bed9479070 RCX: 00007f108289e4ab
Feb 24 06:28:20 pve1 kernel: RDX: 0000000000000001 RSI: 0000000000000081 RDI: 0000000000000026
Feb 24 06:28:20 pve1 kernel: RBP: 00007f1068003500 R08: 0000000000000000 R09: 00000000ffffffff
Feb 24 06:28:20 pve1 kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
Feb 24 06:28:20 pve1 kernel: R13: 00007f1068003578 R14: 000055bed704cb05 R15: 0000000000802000
Feb 24 06:28:20 pve1 kernel: </TASK>
Feb 24 06:30:21 pve1 kernel: INFO: task worker:3212776 blocked for more than 483 seconds.
Feb 24 06:30:21 pve1 kernel: Tainted: P O 5.15.85-1-pve #1
Feb 24 06:30:21 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 24 06:30:21 pve1 kernel: task:worker state:D stack: 0 pid:3212776 ppid: 1 flags:0x00004000
Feb 24 06:30:21 pve1 kernel: Call Trace:
Feb 24 06:30:21 pve1 kernel: <TASK>
Feb 24 06:30:21 pve1 kernel: __schedule+0x34e/0x1740
Feb 24 06:30:21 pve1 kernel: ? release_pages+0x162/0x530
Feb 24 06:30:21 pve1 kernel: schedule+0x69/0x110
Feb 24 06:30:21 pve1 kernel: io_schedule+0x46/0x80
Feb 24 06:30:21 pve1 kernel: wait_on_page_bit_common+0x114/0x3e0
Feb 24 06:30:21 pve1 kernel: ? filemap_invalidate_unlock_two+0x50/0x50
Feb 24 06:30:21 pve1 kernel: wait_on_page_bit+0x3f/0x50
Feb 24 06:30:21 pve1 kernel: wait_on_page_writeback+0x26/0x80
Feb 24 06:30:21 pve1 kernel: __filemap_fdatawait_range+0x97/0x120
Feb 24 06:30:21 pve1 kernel: file_write_and_wait_range+0xd3/0xf0
Feb 24 06:30:21 pve1 kernel: cifs_strict_fsync+0x3b/0x430 [cifs]
Feb 24 06:30:21 pve1 kernel: __x64_sys_fdatasync+0x4c/0x90
Feb 24 06:30:21 pve1 kernel: do_syscall_64+0x5c/0xc0
Feb 24 06:30:21 pve1 kernel: ? syscall_exit_to_user_mode+0x27/0x50
Feb 24 06:30:21 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:30:21 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:30:21 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:30:21 pve1 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Feb 24 06:30:21 pve1 kernel: RIP: 0033:0x7f108289e4ab
Feb 24 06:30:21 pve1 kernel: RSP: 002b:00007f0cf6cf5330 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Feb 24 06:30:21 pve1 kernel: RAX: ffffffffffffffda RBX: 000055bed9479070 RCX: 00007f108289e4ab
Feb 24 06:30:21 pve1 kernel: RDX: 0000000000000001 RSI: 0000000000000081 RDI: 0000000000000026
Feb 24 06:30:21 pve1 kernel: RBP: 00007f1068003500 R08: 0000000000000000 R09: 00000000ffffffff
Feb 24 06:30:21 pve1 kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
Feb 24 06:30:21 pve1 kernel: R13: 00007f1068003578 R14: 000055bed704cb05 R15: 0000000000802000
Feb 24 06:30:21 pve1 kernel: </TASK>
Feb 24 06:32:22 pve1 kernel: INFO: task worker:3212776 blocked for more than 604 seconds.
Feb 24 06:32:22 pve1 kernel: Tainted: P O 5.15.85-1-pve #1
Feb 24 06:32:22 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 24 06:32:22 pve1 kernel: task:worker state:D stack: 0 pid:3212776 ppid: 1 flags:0x00004000
Feb 24 06:32:22 pve1 kernel: Call Trace:
Feb 24 06:32:22 pve1 kernel: <TASK>
Feb 24 06:32:22 pve1 kernel: __schedule+0x34e/0x1740
Feb 24 06:32:22 pve1 kernel: ? release_pages+0x162/0x530
Feb 24 06:32:22 pve1 kernel: schedule+0x69/0x110
Feb 24 06:32:22 pve1 kernel: io_schedule+0x46/0x80
Feb 24 06:32:22 pve1 kernel: wait_on_page_bit_common+0x114/0x3e0
Feb 24 06:32:22 pve1 kernel: ? filemap_invalidate_unlock_two+0x50/0x50
Feb 24 06:32:22 pve1 kernel: wait_on_page_bit+0x3f/0x50
Feb 24 06:32:22 pve1 kernel: wait_on_page_writeback+0x26/0x80
Feb 24 06:32:22 pve1 kernel: __filemap_fdatawait_range+0x97/0x120
Feb 24 06:32:22 pve1 kernel: file_write_and_wait_range+0xd3/0xf0
Feb 24 06:32:22 pve1 kernel: cifs_strict_fsync+0x3b/0x430 [cifs]
Feb 24 06:32:22 pve1 kernel: __x64_sys_fdatasync+0x4c/0x90
Feb 24 06:32:22 pve1 kernel: do_syscall_64+0x5c/0xc0
Feb 24 06:32:22 pve1 kernel: ? syscall_exit_to_user_mode+0x27/0x50
Feb 24 06:32:22 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:32:22 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:32:22 pve1 kernel: ? do_syscall_64+0x69/0xc0
Feb 24 06:32:22 pve1 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Feb 24 06:32:22 pve1 kernel: RIP: 0033:0x7f108289e4ab
Feb 24 06:32:22 pve1 kernel: RSP: 002b:00007f0cf6cf5330 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Feb 24 06:32:22 pve1 kernel: RAX: ffffffffffffffda RBX: 000055bed9479070 RCX: 00007f108289e4ab
Feb 24 06:32:22 pve1 kernel: RDX: 0000000000000001 RSI: 0000000000000081 RDI: 0000000000000026
Feb 24 06:32:22 pve1 kernel: RBP: 00007f1068003500 R08: 0000000000000000 R09: 00000000ffffffff
Feb 24 06:32:22 pve1 kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
Feb 24 06:32:22 pve1 kernel: R13: 00007f1068003578 R14: 000055bed704cb05 R15: 0000000000802000
Feb 24 06:32:22 pve1 kernel: </TASK>