Proxmox high IO freeze on migration, clone or importing VM

[Trust_ITS]

Hi everyone,

I have installed Proxmox VE 7 on a Supermicro server with a LSI RAID controller with 4 disks Hitachi SAS.
When importing a VM from NFS, or cloning a VM, or doing a live migration from another node of the cluster, the VM running on the host freezes.
I just looked at CPU and RAM with top and it seems no problem, instead load average and IO are very high during activity.
In syslog I have the following lines:

May 25 09:17:04 vm7b kernel: INFO: task qemu-img:121476 blocked for more than 483 seconds.
May 25 09:17:04 vm7b kernel:       Tainted: P           O      5.15.107-2-pve #1
May 25 09:17:04 vm7b kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 25 09:17:04 vm7b kernel: task:qemu-img        state stack:    0 pid:121476 ppid:118075 flags:0x00000000
May 25 09:17:04 vm7b kernel: Call Trace:
May 25 09:17:04 vm7b kernel:  <TASK>
May 25 09:17:04 vm7b kernel:  __schedule+0x34e/0x1740
May 25 09:17:04 vm7b kernel:  ? __submit_bio+0x167/0x230
May 25 09:17:04 vm7b kernel:  ? mempool_alloc_slab+0x17/0x20
May 25 09:17:04 vm7b kernel:  ? mempool_alloc+0x64/0x1b0
May 25 09:17:04 vm7b kernel:  ? bit_wait+0x70/0x70
May 25 09:17:04 vm7b kernel:  schedule+0x69/0x110
May 25 09:17:04 vm7b kernel:  io_schedule+0x46/0x80
May 25 09:17:04 vm7b kernel:  bit_wait_io+0x11/0x70
May 25 09:17:04 vm7b kernel:  __wait_on_bit+0x33/0xa0
May 25 09:17:04 vm7b kernel:  out_of_line_wait_on_bit+0x8d/0xb0
May 25 09:17:04 vm7b kernel:  ? var_wake_function+0x30/0x30
May 25 09:17:04 vm7b kernel:  __block_write_begin_int+0x222/0x590
May 25 09:17:04 vm7b kernel:  ? blkdev_llseek+0x70/0x70
May 25 09:17:04 vm7b kernel:  ? blkdev_llseek+0x70/0x70
May 25 09:17:04 vm7b kernel:  block_write_begin+0x4d/0xf0
May 25 09:17:04 vm7b kernel:  blkdev_write_begin+0x23/0x30
May 25 09:17:04 vm7b kernel:  generic_perform_write+0xc6/0x1f0
May 25 09:17:04 vm7b kernel:  __generic_file_write_iter+0x10f/0x1e0
May 25 09:17:04 vm7b kernel:  blkdev_write_iter+0xb3/0x160
May 25 09:17:04 vm7b kernel:  new_sync_write+0x114/0x1b0
May 25 09:17:04 vm7b kernel:  vfs_write+0x1d9/0x270
May 25 09:17:04 vm7b kernel:  __x64_sys_pwrite64+0x97/0xc0
May 25 09:17:04 vm7b kernel:  do_syscall_64+0x5c/0xc0
May 25 09:17:04 vm7b kernel:  ? __x64_sys_futex+0x81/0x1d0
May 25 09:17:04 vm7b kernel:  ? do_syscall_64+0x69/0xc0
May 25 09:17:04 vm7b kernel:  ? exit_to_user_mode_prepare+0x37/0x1b0
May 25 09:17:04 vm7b kernel:  ? syscall_exit_to_user_mode+0x27/0x50
May 25 09:17:04 vm7b kernel:  ? do_syscall_64+0x69/0xc0
May 25 09:17:04 vm7b kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
May 25 09:17:04 vm7b kernel: RIP: 0033:0x7f51d999c9c7
May 25 09:17:04 vm7b kernel: RSP: 002b:00007f51c6df6680 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
May 25 09:17:04 vm7b kernel: RAX: ffffffffffffffda RBX: 00007f51cc1c7000 RCX: 00007f51d999c9c7
May 25 09:17:04 vm7b kernel: RDX: 0000000000200000 RSI: 00007f51cc1c7000 RDI: 0000000000000008
May 25 09:17:04 vm7b kernel: RBP: 00007f51cc4c8370 R08: 0000000000000000 R09: 00000000ffffffff
May 25 09:17:04 vm7b kernel: R10: 000000051affec00 R11: 0000000000000293 R12: 0000000000000000
May 25 09:17:04 vm7b kernel: R13: 000055f8d18bbb08 R14: 000055f8cfd34e75 R15: 0000000000802000
May 25 09:17:04 vm7b kernel:  </TASK>

I tried updating BIOS and LSI RAID controller firmware, tried reinstalling from RAID6 to RAID1, but the problem persists.
In addition, my RAID BBU battery is gone, I don't know if it could be the problem.

I would like try installing with 6.2 pve kernel, but then I'll have to upgrade the other node of the cluster that is working fine.
Any suggestion could be really helpful.
Thanks

 

[floh8]

to change the kernel is a good idea, i think. u dont have to upgrade the other node.

 

[Trust_ITS]

to change the kernel is a good idea, i think. u dont have to upgrade the other node.

So I'll try with 6.2 and report back here

 

[Trust_ITS]

With 6.2 kernel I have timeout too, following syslog error during a VM import:
pvestatd[1036]: VM 110 qmp command failed - VM 110 qmp command 'query-proxmox-support' failed - unable to connect to VM 110 qmp socket - timeout after 51 retries
I think it's something related to LSI RAID controller or HDs but I don't know how to fix this ...