Hung on restore since Upgrade to Kernel 7 / Proxmox 9

A

aklj_ekj

New Member
Oct 12, 2024
8
0
1
Note: Already posted on https://forum.proxmox.com/threads/server-disk-i-o-delay-100-during-cloning-and-backup.173051/page-3

Hey,
I seem to have the same problem since I've upgraded from Proxmox 8 to 9 - or at least something similar (currently running Kernel 7.0.2-4-pve).
My system freezes when I clone or restore a VM. No problem with backups or while running the system.
Freeze:
- I still can access the web gui, sometimes all VM States are not populated (question mark)
- I can access the server trough SSH, sometime also via webshell but not always.
- Reboot can be initiated via web gui or SSH but will hang somewhere in the process, need to reboot as shown below.
- I've just retried today to restore a 80GB backup - the restore process was at 100% within ~2 minutes but hung after that - I had to reboot the server after ~30minutes no progress

Mostly the clone/restore works until 100% and THEN the systems starts to hang before the log shows TASK OK.

The other VMs start to freeze and I cannot reboot the server - I have to use the following commands via SSH/Shell to reboot the server:
Code: 
echo 1 > /proc/sys/kernel/sysrq
echo s > /proc/sysrq-trigger
sleep 2
echo u > /proc/sysrq-trigger
sleep 2
echo b > /proc/sysrq-trigger

Could it have to do something with this?
https://bugzilla.proxmox.com/show_bug.cgi?id=7052

Hardware: ProLiant DL360 G7
RAID: HP Smart Array G6 with Spinning discs (one Array for system and VM), ssacli shows all disks OK
Configuration:
- System and VMs on LVM Thin



Addition:
I had the same or similar problem on another server, this one locked itself up multiple times.
I was able to resolve the self lock up by moving the VM Disks to another RAID on the same server (still spinning discs)

Logfile excempts:
iostat -xz 1 shows large w_await (>290000 ms), and 100% utilization


Journalctl:
Code: 
May 13 17:44:47 host kernel: INFO: task iou-wrk-1650:1748 blocked for more than 122 seconds.
May 13 17:44:47 host kernel:       Tainted: P          IO        7.0.2-2-pve #1
May 13 17:44:47 host kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 13 17:44:47 host kernel: task:iou-wrk-1650    state:D stack:0     pid:1748  tgid:1650  ppid:1      task_flags:0x84040d0 flags:0x00080000
May 13 17:44:47 host kernel: Call Trace:
May 13 17:44:47 host kernel:  <TASK>
May 13 17:44:47 host kernel:  __schedule+0x495/0x1760
May 13 17:44:47 host kernel:  ? __blk_flush_plug+0xef/0x150
May 13 17:44:47 host kernel:  schedule+0x27/0xf0
May 13 17:44:47 host kernel:  io_schedule+0x4c/0x80
May 13 17:44:47 host kernel:  folio_wait_bit_common+0x136/0x340
May 13 17:44:47 host kernel:  ? __pfx_wake_page_function+0x10/0x10
May 13 17:44:47 host kernel:  folio_wait_bit+0x18/0x30
May 13 17:44:47 host kernel:  folio_wait_writeback+0x3d/0xb0
May 13 17:44:47 host kernel:  writeback_iter+0xda/0x310
May 13 17:44:47 host kernel:  blkdev_writepages+0x7f/0xd0
May 13 17:44:47 host kernel:  do_writepages+0xc4/0x180
May 13 17:44:47 host kernel:  filemap_writeback+0xd1/0x100
May 13 17:44:47 host kernel:  file_write_and_wait_range+0x60/0xd0
May 13 17:44:47 host kernel:  blkdev_fsync+0x36/0x60
May 13 17:44:47 host kernel:  vfs_fsync_range+0x2d/0xa0
May 13 17:44:47 host kernel:  io_fsync+0x3d/0x60
May 13 17:44:47 host kernel:  __io_issue_sqe+0x43/0x1b0
May 13 17:44:47 host kernel:  io_issue_sqe+0x3e/0x5b0
May 13 17:44:47 host kernel:  io_wq_submit_work+0xdf/0x380
May 13 17:44:47 host kernel:  io_worker_handle_work+0x13d/0x570
May 13 17:44:47 host kernel:  io_wq_worker+0x101/0x3b0
May 13 17:44:47 host kernel:  ? raw_spin_rq_unlock+0x14/0x50
May 13 17:44:47 host kernel:  ? finish_task_switch.isra.0+0x95/0x2f0
May 13 17:44:47 host kernel:  ? __pfx_io_wq_worker+0x10/0x10
May 13 17:44:47 host kernel:  ret_from_fork+0x2dc/0x3a0
May 13 17:44:47 host kernel:  ? __pfx_io_wq_worker+0x10/0x10
May 13 17:44:47 host kernel:  ret_from_fork_asm+0x1a/0x30
May 13 17:44:47 host kernel: RIP: 0033:0x0
May 13 17:44:47 host kernel: RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
May 13 17:44:47 host kernel: RAX: 0000000000000000 RBX: 00005b32e77b52d8 RCX: 00007c13d6ce63ca
May 13 17:44:47 host kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000000000001a
May 13 17:44:47 host kernel: RBP: 00005b32e77b53c0 R08: 0000000000000000 R09: 0000000000000008
May 13 17:44:47 host kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005b32e77b52d0
May 13 17:44:47 host kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
May 13 17:44:47 host kernel:  </TASK>


May 13 19:19:16 host kernel: INFO: task worker:1889 blocked for more than 122 seconds.
May 13 19:19:16 host kernel:       Tainted: P          IO        7.0.2-2-pve #1
May 13 19:19:16 host kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 13 19:19:16 host kernel: task:worker          state:D stack:0     pid:1889  tgid:1881  ppid:1854   task_flags:0x400040 flags:0x00080000
May 13 19:19:16 host kernel: Call Trace:
May 13 19:19:16 host kernel:  <TASK>
May 13 19:19:16 host kernel:  __schedule+0x495/0x1760
May 13 19:19:16 host kernel:  ? __submit_bio+0x196/0x250
May 13 19:19:16 host kernel:  ? __pfx_bit_wait_io+0x10/0x10
May 13 19:19:16 host kernel:  schedule+0x27/0xf0
May 13 19:19:16 host kernel:  io_schedule+0x4c/0x80
May 13 19:19:16 host kernel:  bit_wait_io+0x11/0x80
May 13 19:19:16 host kernel:  __wait_on_bit+0x34/0xa0
May 13 19:19:16 host kernel:  out_of_line_wait_on_bit+0x8d/0xc0
May 13 19:19:16 host kernel:  ? __pfx_wake_bit_function+0x10/0x10
May 13 19:19:16 host kernel:  __block_write_begin_int+0x24f/0x560
May 13 19:19:16 host kernel:  iomap_write_begin+0x4cf/0x790
May 13 19:19:16 host kernel:  ? radix_tree_lookup+0xd/0x20
May 13 19:19:16 host kernel:  iomap_file_buffered_write+0x1f8/0x4a0
May 13 19:19:16 host kernel:  blkdev_write_iter+0x192/0x350
May 13 19:19:16 host kernel:  ? rw_verify_area+0x57/0x190
May 13 19:19:16 host kernel:  vfs_write+0x274/0x490
May 13 19:19:16 host kernel:  __x64_sys_pwrite64+0x98/0xd0
May 13 19:19:16 host kernel:  x64_sys_call+0x1d12/0x2390
May 13 19:19:16 host kernel:  do_syscall_64+0x11c/0x14e0
May 13 19:19:16 host kernel:  ? do_syscall_64+0x311/0x14e0
May 13 19:19:16 host kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
May 13 19:19:16 host kernel: RIP: 0033:0x7341e8ea69ee
May 13 19:19:16 host kernel: RSP: 002b:00007341dd7f5f28 EFLAGS: 00000246 ORIG_RAX: 0000000000000012
May 13 19:19:16 host kernel: RAX: ffffffffffffffda RBX: 00007341dd7fa6c0 RCX: 00007341e8ea69ee
May 13 19:19:16 host kernel: RDX: 0000000000200000 RSI: 00007341e4e3a000 RDI: 000000000000000a
May 13 19:19:16 host kernel: RBP: 00007341e4e3a000 R08: 0000000000000000 R09: 0000000000000000
May 13 19:19:16 host kernel: R10: 00000000db1ffe00 R11: 0000000000000246 R12: 0000000000000000
May 13 19:19:16 host kernel: R13: 00005b76c37f41de R14: 00005b76fb84cf58 R15: 00007341dcffa000
May 13 19:19:16 host kernel:  </TASK>
 
Last edited:
aklj_ekj said:
Hey,
I seem to have the same problem since I've upgraded from Proxmox 8 to 9 - or at least something similar (currently running Kernel 7.0.2-4-pve).
My system freezes when I clone or restore a VM. No problem with backups or while running the system.
Freeze:
- I still can access the web gui, sometimes all VM States are not populated (question mark)
- I can access the server trough SSH, sometime also via webshell but not always.
- Reboot can be initiated via web gui or SSH but will hang somewhere in the process, need to reboot as shown below.
- I've just retried today to restore a 80GB backup - the restore process was at 100% within ~2 minutes but hung after that - I had to reboot the server after ~30minutes no progress
Click to expand...

I just posted that warning here regarding my PBS on my Synology which completely went south after the last update (introducing Kernel 7): https://forum.proxmox.com/threads/hung-on-restore-since-upgrade-to-kernel-7-proxmox-9.183717/

Your issue seems very similar, also pointing to the new io_uring in Kernel 7. Your fsync worker seems to hang here in your Kernel Stack-Trace. So that issue seems not to be related only to my (unsupported) PBS-VM in Synology DSM, it happens also in PVE on bare metal.

I guess (hope) the Proxmox Dev's will have an eye on that and fix this issue. Meanwhile I'll pin Kernel 6 on all of my hosts right now to avoid unwanted suprises after the next maintenance reboot.

Take care,
Marco
 
I just had to restore a VM from a backup yesterday - still the same thing.

First try under Kernel 7 with Proxmox 9.2:

Code: 
- Restore start 19:41:10, Kernel 7.0.6-2 pve -

Jun 16 19:46:07 hostn pvedaemon[3710013]: <root@pam> starting task UPID:proxmox:XXXXXXXX:XXXXXXXX:XXXXXXXX:qmrestore:203:root@pam:
...
Jun 16 19:49:38 hostn kernel: INFO: task iou-wrk-1932:3707137 blocked for more than 122 seconds.
Jun 16 19:49:38 hostn kernel:       Tainted: P          IO        7.0.2-6-pve #1
Jun 16 19:49:38 hostn kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 16 19:49:38 hostn kernel: task:iou-wrk-1932    state:D stack:0     pid:3707137 tgid:1930  ppid:1      task_flags:0x84040d0 flags:0x00080000
Jun 16 19:49:38 hostn kernel: Call Trace:
Jun 16 19:49:38 hostn kernel:  <TASK>
Jun 16 19:49:38 hostn kernel:  __schedule+0x495/0x1760
Jun 16 19:49:38 hostn kernel:  ? __wb_update_bandwidth+0x141/0x220
Jun 16 19:49:38 hostn kernel:  schedule+0x27/0xf0
Jun 16 19:49:38 hostn kernel:  io_schedule+0x4c/0x80
Jun 16 19:49:38 hostn kernel:  folio_wait_bit_common+0x136/0x340
Jun 16 19:49:38 hostn kernel:  ? __pfx_wake_page_function+0x10/0x10
Jun 16 19:49:38 hostn kernel:  folio_wait_bit+0x18/0x30
Jun 16 19:49:38 hostn kernel:  folio_wait_writeback+0x3d/0xb0
Jun 16 19:49:38 hostn kernel:  __filemap_fdatawait_range+0x87/0xf0
Jun 16 19:49:38 hostn kernel:  file_write_and_wait_range+0x99/0xd0
Jun 16 19:49:38 hostn kernel:  blkdev_fsync+0x36/0x60
Jun 16 19:49:38 hostn kernel:  vfs_fsync_range+0x2d/0xa0
Jun 16 19:49:38 hostn kernel:  io_fsync+0x3d/0x60
Jun 16 19:49:38 hostn kernel:  __io_issue_sqe+0x43/0x1b0
Jun 16 19:49:38 hostn kernel:  io_issue_sqe+0x3e/0x5b0
Jun 16 19:49:38 hostn kernel:  io_wq_submit_work+0xdf/0x380
Jun 16 19:49:38 hostn kernel:  io_worker_handle_work+0x13d/0x570
Jun 16 19:49:38 hostn kernel:  io_wq_worker+0x101/0x3b0
Jun 16 19:49:38 hostn kernel:  ? raw_spin_rq_unlock+0x14/0x50
Jun 16 19:49:38 hostn kernel:  ? finish_task_switch.isra.0+0x95/0x2f0
Jun 16 19:49:38 hostn kernel:  ? __pfx_io_wq_worker+0x10/0x10
Jun 16 19:49:38 hostn kernel:  ret_from_fork+0x2dc/0x3a0
Jun 16 19:49:38 hostn kernel:  ? __pfx_io_wq_worker+0x10/0x10
Jun 16 19:49:38 hostn kernel:  ret_from_fork_asm+0x1a/0x30
Jun 16 19:49:38 hostn kernel: RIP: 0033:0x0
Jun 16 19:49:38 hostn kernel: RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
Jun 16 19:49:38 hostn kernel: RAX: 0000000000000000 RBX: 000056665edceea0 RCX: 000074f0216883ca
Jun 16 19:49:38 hostn kernel: RDX: 0000000000000000 RSI: 0000000000000004 RDI: 000000000000000c
Jun 16 19:49:38 hostn kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000008
Jun 16 19:49:38 hostn kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000074f01b674ca0
Jun 16 19:49:38 hostn kernel: R13: 000056665edceda0 R14: 00066d24d5522f1f R15: 0000000000000000
Jun 16 19:49:38 hostn kernel:  </TASK>
Jun 16 19:51:41 hostn kernel: INFO: task iou-wrk-1932:3707137 blocked for more than 245 seconds.
Jun 16 19:51:41 hostn kernel:       Tainted: P          IO        7.0.2-6-pve #1
Jun 16 19:51:41 hostn kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 16 19:51:41 hostn kernel: task:iou-wrk-1932    state:D stack:0     pid:3707137 tgid:1930  ppid:1      task_flags:0x84040d0 flags:0x00080000
Jun 16 19:51:41 hostn kernel: Call Trace:
Jun 16 19:51:41 hostn kernel:  <TASK>
Jun 16 19:51:41 hostn kernel:  __schedule+0x495/0x1760
Jun 16 19:51:41 hostn kernel:  ? __wb_update_bandwidth+0x141/0x220
Jun 16 19:51:41 hostn kernel:  schedule+0x27/0xf0
Jun 16 19:51:41 hostn kernel:  io_schedule+0x4c/0x80
Jun 16 19:51:41 hostn kernel:  folio_wait_bit_common+0x136/0x340
Jun 16 19:51:41 hostn kernel:  ? __pfx_wake_page_function+0x10/0x10
Jun 16 19:51:41 hostn kernel:  folio_wait_bit+0x18/0x30
Jun 16 19:51:41 hostn kernel:  folio_wait_writeback+0x3d/0xb0
Jun 16 19:51:41 hostn kernel:  __filemap_fdatawait_range+0x87/0xf0
Jun 16 19:51:41 hostn kernel:  file_write_and_wait_range+0x99/0xd0
Jun 16 19:51:41 hostn kernel:  blkdev_fsync+0x36/0x60
Jun 16 19:51:41 hostn kernel:  vfs_fsync_range+0x2d/0xa0
Jun 16 19:51:41 hostn kernel:  io_fsync+0x3d/0x60
Jun 16 19:51:41 hostn kernel:  __io_issue_sqe+0x43/0x1b0
Jun 16 19:51:41 hostn kernel:  io_issue_sqe+0x3e/0x5b0
Jun 16 19:51:41 hostn kernel:  io_wq_submit_work+0xdf/0x380
Jun 16 19:51:41 hostn kernel:  io_worker_handle_work+0x13d/0x570
Jun 16 19:51:41 hostn kernel:  io_wq_worker+0x101/0x3b0
Jun 16 19:51:41 hostn kernel:  ? raw_spin_rq_unlock+0x14/0x50
Jun 16 19:51:41 hostn kernel:  ? finish_task_switch.isra.0+0x95/0x2f0
Jun 16 19:51:41 hostn kernel:  ? __pfx_io_wq_worker+0x10/0x10
Jun 16 19:51:41 hostn kernel:  ret_from_fork+0x2dc/0x3a0
Jun 16 19:51:41 hostn kernel:  ? __pfx_io_wq_worker+0x10/0x10
Jun 16 19:51:41 hostn kernel:  ret_from_fork_asm+0x1a/0x30
Jun 16 19:51:41 hostn kernel: RIP: 0033:0x0
Jun 16 19:51:41 hostn kernel: RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
Jun 16 19:51:41 hostn kernel: RAX: 0000000000000000 RBX: 000056665edceea0 RCX: 000074f0216883ca
Jun 16 19:51:41 hostn kernel: RDX: 0000000000000000 RSI: 0000000000000004 RDI: 000000000000000c
Jun 16 19:51:41 hostn kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000008
Jun 16 19:51:41 hostn kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000074f01b674ca0
Jun 16 19:51:41 hostn kernel: R13: 000056665edceda0 R14: 00066d24d5522f1f R15: 0000000000000000
Jun 16 19:51:41 hostn kernel:  </TASK>
Jun 16 19:53:43 hostn kernel: INFO: task iou-wrk-1932:3707137 blocked for more than 368 seconds.
Jun 16 19:53:43 hostn kernel:       Tainted: P          IO        7.0.2-6-pve #1
Jun 16 19:53:43 hostn kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 16 19:53:43 hostn kernel: task:iou-wrk-1932    state:D stack:0     pid:3707137 tgid:1930  ppid:1      task_flags:0x84040d0 flags:0x00080000
Jun 16 19:53:43 hostn kernel: Call Trace:
Jun 16 19:53:43 hostn kernel:  <TASK>
Jun 16 19:53:43 hostn kernel:  __schedule+0x495/0x1760
Jun 16 19:53:43 hostn kernel:  ? __wb_update_bandwidth+0x141/0x220
Jun 16 19:53:43 hostn kernel:  schedule+0x27/0xf0
Jun 16 19:53:43 hostn kernel:  io_schedule+0x4c/0x80
Jun 16 19:53:43 hostn kernel:  folio_wait_bit_common+0x136/0x340
Jun 16 19:53:43 hostn kernel:  ? __pfx_wake_page_function+0x10/0x10
Jun 16 19:53:43 hostn kernel:  folio_wait_bit+0x18/0x30
Jun 16 19:53:43 hostn kernel:  folio_wait_writeback+0x3d/0xb0
Jun 16 19:53:43 hostn kernel:  __filemap_fdatawait_range+0x87/0xf0
Jun 16 19:53:43 hostn kernel:  file_write_and_wait_range+0x99/0xd0
Jun 16 19:53:43 hostn kernel:  blkdev_fsync+0x36/0x60
Jun 16 19:53:43 hostn kernel:  vfs_fsync_range+0x2d/0xa0
Jun 16 19:53:43 hostn kernel:  io_fsync+0x3d/0x60
Jun 16 19:53:43 hostn kernel:  __io_issue_sqe+0x43/0x1b0
Jun 16 19:53:43 hostn kernel:  io_issue_sqe+0x3e/0x5b0
Jun 16 19:53:43 hostn kernel:  io_wq_submit_work+0xdf/0x380
Jun 16 19:53:43 hostn kernel:  io_worker_handle_work+0x13d/0x570
Jun 16 19:53:43 hostn kernel:  io_wq_worker+0x101/0x3b0
Jun 16 19:53:43 hostn kernel:  ? raw_spin_rq_unlock+0x14/0x50
Jun 16 19:53:43 hostn kernel:  ? finish_task_switch.isra.0+0x95/0x2f0
Jun 16 19:53:43 hostn kernel:  ? __pfx_io_wq_worker+0x10/0x10
Jun 16 19:53:43 hostn kernel:  ret_from_fork+0x2dc/0x3a0
Jun 16 19:53:43 hostn kernel:  ? __pfx_io_wq_worker+0x10/0x10
Jun 16 19:53:43 hostn kernel:  ret_from_fork_asm+0x1a/0x30
Jun 16 19:53:43 hostn kernel: RIP: 0033:0x0
Jun 16 19:53:43 hostn kernel: RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
Jun 16 19:53:43 hostn kernel: RAX: 0000000000000000 RBX: 000056665edceea0 RCX: 000074f0216883ca
Jun 16 19:53:43 hostn kernel: RDX: 0000000000000000 RSI: 0000000000000004 RDI: 000000000000000c
Jun 16 19:53:43 hostn kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000008
Jun 16 19:53:43 hostn kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000074f01b674ca0
Jun 16 19:53:43 hostn kernel: R13: 000056665edceda0 R14: 00066d24d5522f1f R15: 0000000000000000
Jun 16 19:53:43 hostn kernel:  </TASK>
... multiple repeats of the above error ...
Jun 16 20:03:58 hostn kernel: INFO: task iou-wrk-1932:3707137 blocked for more than 983 seconds.
Jun 16 20:03:58 hostn kernel:       Tainted: P          IO        7.0.2-6-pve #1
Jun 16 20:03:58 hostn kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 16 20:03:58 hostn kernel: task:iou-wrk-1932    state:D stack:0     pid:3707137 tgid:1930  ppid:1      task_flags:0x84040d0 flags:0x00080000
Jun 16 20:03:58 hostn kernel: Call Trace:
Jun 16 20:03:58 hostn kernel:  <TASK>
Jun 16 20:03:58 hostn kernel:  __schedule+0x495/0x1760
Jun 16 20:03:58 hostn kernel:  ? __wb_update_bandwidth+0x141/0x220
Jun 16 20:03:58 hostn kernel:  schedule+0x27/0xf0
Jun 16 20:03:58 hostn kernel:  io_schedule+0x4c/0x80
Jun 16 20:03:58 hostn kernel:  folio_wait_bit_common+0x136/0x340
Jun 16 20:03:58 hostn kernel:  ? __pfx_wake_page_function+0x10/0x10
Jun 16 20:03:58 hostn kernel:  folio_wait_bit+0x18/0x30
Jun 16 20:03:58 hostn kernel:  folio_wait_writeback+0x3d/0xb0
Jun 16 20:03:58 hostn kernel:  __filemap_fdatawait_range+0x87/0xf0
Jun 16 20:03:58 hostn kernel:  file_write_and_wait_range+0x99/0xd0
Jun 16 20:03:58 hostn kernel:  blkdev_fsync+0x36/0x60
Jun 16 20:03:58 hostn kernel:  vfs_fsync_range+0x2d/0xa0
Jun 16 20:03:58 hostn kernel:  io_fsync+0x3d/0x60
Jun 16 20:03:58 hostn kernel:  __io_issue_sqe+0x43/0x1b0
Jun 16 20:03:58 hostn kernel:  io_issue_sqe+0x3e/0x5b0
Jun 16 20:03:58 hostn kernel:  io_wq_submit_work+0xdf/0x380
Jun 16 20:03:58 hostn kernel:  io_worker_handle_work+0x13d/0x570
Jun 16 20:03:58 hostn kernel:  io_wq_worker+0x101/0x3b0
Jun 16 20:03:58 hostn kernel:  ? raw_spin_rq_unlock+0x14/0x50
Jun 16 20:03:58 hostn kernel:  ? finish_task_switch.isra.0+0x95/0x2f0
Jun 16 20:03:58 hostn kernel:  ? __pfx_io_wq_worker+0x10/0x10
Jun 16 20:03:58 hostn kernel:  ret_from_fork+0x2dc/0x3a0
Jun 16 20:03:58 hostn kernel:  ? __pfx_io_wq_worker+0x10/0x10
Jun 16 20:03:58 hostn kernel:  ret_from_fork_asm+0x1a/0x30
Jun 16 20:03:58 hostn kernel: RIP: 0033:0x0
Jun 16 20:03:58 hostn kernel: RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
Jun 16 20:03:58 hostn kernel: RAX: 0000000000000000 RBX: 000056665edceea0 RCX: 000074f0216883ca
Jun 16 20:03:58 hostn kernel: RDX: 0000000000000000 RSI: 0000000000000004 RDI: 000000000000000c
Jun 16 20:03:58 hostn kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000008
Jun 16 20:03:58 hostn kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000074f01b674ca0
Jun 16 20:03:58 hostn kernel: R13: 000056665edceda0 R14: 00066d24d5522f1f R15: 0000000000000000
Jun 16 20:03:58 hostn kernel:  </TASK>
... trying to shutdown other running VMs and rebooting ...
Jun 16 20:11:06 hostn kernel: ------------[ cut here ]------------
Jun 16 20:11:06 hostn kernel: WARNING: io_uring/io_uring.c:2345 at io_ring_exit_work+0x128/0x4a6, CPU#5: kworker/u65:3/3698409
Jun 16 20:11:06 hostn kernel: Modules linked in: tcp_diag inet_diag veth rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace netfs ebtable_filter ebtables ip6table_raw ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables iptable_raw xt>
Jun 16 20:11:06 hostn kernel:  vfio_iommu_type1 vfio iommufd efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs libblake2b xor raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio hid_generic usbkbd usbmouse us>
Jun 16 20:11:06 hostn kernel: CPU: 5 UID: 0 PID: 3698409 Comm: kworker/u65:3 Tainted: P          IO        7.0.2-6-pve #1 PREEMPT(lazy)
Jun 16 20:11:06 hostn kernel: Tainted: [P]=PROPRIETARY_MODULE, [I]=FIRMWARE_WORKAROUND, [O]=OOT_MODULE
Jun 16 20:11:06 hostn kernel: Hardware name: HP ProLiant -- Redacted
Jun 16 20:11:06 hostn kernel: Workqueue: iou_exit io_ring_exit_work
Jun 16 20:11:06 hostn kernel: RIP: 0010:io_ring_exit_work+0x128/0x4a6
Jun 16 20:11:06 hostn kernel: Code: 8b e8 2c 8c 97 00 4c 89 f7 e8 64 c4 96 00 4c 89 e7 e8 6c f9 ff ff 31 ff 48 89 7c 24 10 48 8b 05 ce dc 39 02 48 39 04 24 79 08 <0f> 0b 41 bd 60 ea 00 00 4c 89 ee 4c 89 ff e8 75 2c 36 01 48 85 c0
Jun 16 20:11:06 hostn kernel: RSP: 0018:ffffcebb4077fda0 EFLAGS: 00010297
Jun 16 20:11:06 hostn kernel: RAX: 000000016c02b839 RBX: ffff8d1620341450 RCX: 0000000000000000
Jun 16 20:11:06 hostn kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Jun 16 20:11:06 hostn kernel: RBP: ffffcebb4077fe30 R08: 0000000000000000 R09: 0000000000000000
Jun 16 20:11:06 hostn kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d1620341000
Jun 16 20:11:06 hostn kernel: R13: 0000000000000032 R14: 0000000000000000 R15: ffff8d1620341470
Jun 16 20:11:06 hostn kernel: FS:  0000000000000000(0000) GS:ffff8d590100f000(0000) knlGS:0000000000000000
Jun 16 20:11:06 hostn kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 16 20:11:06 hostn kernel: CR2: 000057f547941000 CR3: 00000042f083e005 CR4: 00000000000226f0
Jun 16 20:11:06 hostn kernel: Call Trace:
Jun 16 20:11:06 hostn kernel:  <TASK>
Jun 16 20:11:06 hostn kernel:  ? __schedule+0x49d/0x1760
Jun 16 20:11:06 hostn kernel:  process_one_work+0x1a9/0x3c0
Jun 16 20:11:06 hostn kernel:  worker_thread+0x1b8/0x360
Jun 16 20:11:06 hostn kernel:  ? __pfx_worker_thread+0x10/0x10
Jun 16 20:11:06 hostn kernel:  kthread+0xf7/0x130
Jun 16 20:11:06 hostn kernel:  ? __pfx_kthread+0x10/0x10
Jun 16 20:11:06 hostn kernel:  ret_from_fork+0x2dc/0x3a0
Jun 16 20:11:06 hostn kernel:  ? __pfx_kthread+0x10/0x10
Jun 16 20:11:06 hostn kernel:  ret_from_fork_asm+0x1a/0x30
Jun 16 20:11:06 hostn kernel:  </TASK>
Jun 16 20:11:06 hostn kernel: ---[ end trace 0000000000000000 ]---
... multiple daemon stop messages ...
Jun 16 20:11:46 hostn iscsid[1199]: iscsid shutting down.
Jun 16 20:11:46 hostn systemd[1]: iscsid.service: Deactivated successfully.
Jun 16 20:11:46 hostn systemd[1]: Stopped iscsid.service - iSCSI initiator daemon (iscsid).
Jun 16 20:11:46 hostn systemd[1]: iscsid.service: Consumed 45.678s CPU time, 5.9M memory peak.
Jun 16 20:11:46 hostn systemd[1]: Stopped target network-online.target - Network is Online.
Jun 16 20:12:34 hostn kernel: sysrq: Emergency Sync
Jun 16 20:12:36 hostn kernel: sysrq: Emergency Remount R/O
... sysrq reboot ...

Then pinned Kernel 6.8.12-22-pve, rebooted and started the restore again:

Code: 
-- restore start, 2026-06-16 21:17:38
Jun 16 21:17:38 hostn pvedaemon[1440]: <root@pam> starting task UPID:proxmox:XXXXXXXX:XXXXXXXX:XXXXXXXX:qmrestore:203:root@pam:
...
Jun 16 21:25:45 hostn pvedaemon[1440]: <root@pam> end task UPID:proxmox:XXXXXXXX:XXXXXXXX:XXXXXXXX::qmrestore:203:root@pam: OK
-- restore end, 2026-06-16 21:25:45



The system and the vms live on a lvm-thin storage on a HP Smart Array P410i (RAID controlled by controller - no ZFS).



iostat -xz 1 shows 100% utilization in both cases but it seems that there is some sort of deadlock in kernel >=7 that blocks the restore.

As you can see with kernel version 6.8 the restore ran trough within 8 minutes. I've had to cancel the first try after 30 minutes.



An interesting point I've seen yesterday - the 100% disk utilization (iostat) does not go away even if you STOP the restore process.
 
You must log in or register to reply here.
Top Bottom
Back