After 100% read while restoring it gets stuck/freeze, any version, vm, proxmox.

[scroll_dp]

Hi, I have problem with restoring my vms into new proxmox 8 and now I am on 7.4 version, but there is no problem with versions, because even when I wanted to try restore on original proxmox, where backup was done, it never finish. I start restore option, than it reads to 100%, than all qm get stuck, they freezed, IO get to the 8 %, and I can wait 10,20 min, but nothing happed, I try it with small VM with 9 GB of size of disk, but every time, with every backup to restore, with different compression, with different version of proxmox, but same hardware, this occure. Some time ago, it works fine, maybe around 5 months ago with same proxmox installation. I will appreciate any help, thanks.
HereI I send log output, and there is mabye kernel error, but this is first time when it show up, only with no compression on backup:

machine is: hp proliant dl360e

Dec 25 22:33:25 pveProliant pvedaemon[1472]: <root@pam> starting task UPIDveProliant:0000119B:0000C404:6589F525:qmrestore:160:root@pam:
Dec 25 22:35:25 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
Dec 25 22:35:25 pveProliant pvestatd[1442]: status update time (8.228 seconds)
Dec 25 22:35:35 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:35:35 pveProliant pvestatd[1442]: status update time (8.293 seconds)
Dec 25 22:35:45 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:35:46 pveProliant pvestatd[1442]: status update time (8.286 seconds)
Dec 25 22:35:55 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:35:55 pveProliant pvestatd[1442]: status update time (8.267 seconds)
Dec 25 22:36:05 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:36:05 pveProliant pvestatd[1442]: status update time (8.296 seconds)
Dec 25 22:36:15 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:36:15 pveProliant pvestatd[1442]: status update time (8.275 seconds)
Dec 25 22:36:25 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:36:26 pveProliant pvestatd[1442]: status update time (8.284 seconds)
Dec 25 22:36:35 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:36:35 pveProliant pvestatd[1442]: status update time (8.275 seconds)
Dec 25 22:36:45 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:36:45 pveProliant pvestatd[1442]: status update time (8.315 seconds)
Dec 25 22:36:55 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:36:56 pveProliant pvestatd[1442]: status update time (8.275 seconds)
Dec 25 22:37:05 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:37:05 pveProliant pvestatd[1442]: status update time (8.280 seconds)
Dec 25 22:37:09 pveProliant kernel: INFO: task vma:4580 blocked for more than 120 seconds.
Dec 25 22:37:09 pveProliant kernel: Tainted: P IO 5.15.102-1-pve #1
Dec 25 22:37:09 pveProliant kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 25 22:37:09 pveProliant kernel: task:vma state stack: 0 pid: 4580 ppid: 4507 flags:0x00004000
Dec 25 22:37:09 pveProliant kernel: Call Trace:
Dec 25 22:37:09 pveProliant kernel: <TASK>
Dec 25 22:37:09 pveProliant kernel: __schedule+0x34e/0x1740
Dec 25 22:37:09 pveProliant kernel: ? __pagevec_release+0x30/0x70
Dec 25 22:37:09 pveProliant kernel: ? __cond_resched+0x1a/0x50
Dec 25 22:37:09 pveProliant kernel: ? write_cache_pages+0x24d/0x460
Dec 25 22:37:09 pveProliant kernel: ? __set_page_dirty_no_writeback+0x50/0x50
Dec 25 22:37:09 pveProliant kernel: schedule+0x69/0x110
Dec 25 22:37:09 pveProliant kernel: io_schedule+0x46/0x80
Dec 25 22:37:09 pveProliant kernel: wait_on_page_bit_common+0x114/0x3e0
Dec 25 22:37:09 pveProliant kernel: ? filemap_invalidate_unlock_two+0x50/0x50
Dec 25 22:37:09 pveProliant kernel: wait_on_page_bit+0x3f/0x50
Dec 25 22:37:09 pveProliant kernel: wait_on_page_writeback+0x26/0x80
Dec 25 22:37:09 pveProliant kernel: __filemap_fdatawait_range+0x97/0x120
Dec 25 22:37:09 pveProliant kernel: file_write_and_wait_range+0xd3/0xf0
Dec 25 22:37:09 pveProliant kernel: blkdev_fsync+0x3a/0x70
Dec 25 22:37:09 pveProliant kernel: __x64_sys_fdatasync+0x4c/0x90
Dec 25 22:37:09 pveProliant kernel: do_syscall_64+0x5c/0xc0
Dec 25 22:37:09 pveProliant kernel: ? syscall_exit_to_user_mode+0x27/0x50
Dec 25 22:37:09 pveProliant kernel: ? do_syscall_64+0x69/0xc0
Dec 25 22:37:09 pveProliant kernel: ? do_syscall_64+0x69/0xc0
Dec 25 22:37:09 pveProliant kernel: ? do_syscall_64+0x69/0xc0
Dec 25 22:37:09 pveProliant kernel: ? sysvec_apic_timer_interrupt+0x4e/0x90
Dec 25 22:37:09 pveProliant kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 25 22:37:09 pveProliant kernel: RIP: 0033:0x7f4a7d0644ab
Dec 25 22:37:09 pveProliant kernel: RSP: 002b:00007f4a727a96b0 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Dec 25 22:37:09 pveProliant kernel: RAX: ffffffffffffffda RBX: 00005624ce3eced0 RCX: 00007f4a7d0644ab
Dec 25 22:37:09 pveProliant kernel: RDX: 0000000000000001 RSI: 0000000000000081 RDI: 0000000000000009
Dec 25 22:37:09 pveProliant kernel: RBP: 00005624ce3b1a90 R08: 0000000000000000 R09: 00000000ffffffff
Dec 25 22:37:09 pveProliant kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
Dec 25 22:37:09 pveProliant kernel: R13: 00005624ce3b1b08 R14: 00005624ccbb5fc5 R15: 0000000000802000
Dec 25 22:37:09 pveProliant kernel: </TASK>
Dec 25 22:37:15 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:37:15 pveProliant pvestatd[1442]: status update time (8.280 seconds)
Dec 25 22:37:25 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:37:25 pveProliant pvestatd[1442]: status update time (8.283 seconds)
Dec 25 22:37:35 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:37:36 pveProliant pvestatd[1442]: status update time (8.283 seconds)
Dec 25 22:37:45 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:37:45 pveProliant pvestatd[1442]: status update time (8.272 seconds)
Dec 25 22:37:55 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:37:55 pveProliant pvestatd[1442]: status update time (8.293 seconds)
Dec 25 22:38:05 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:38:06 pveProliant pvestatd[1442]: status update time (8.279 seconds)
Dec 25 22:38:15 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:38:15 pveProliant pvestatd[1442]: status update time (8.255 seconds)
Dec 25 22:38:25 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:38:25 pveProliant pvestatd[1442]: status update time (8.272 seconds)
Dec 25 22:38:35 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:38:35 pveProliant pvestatd[1442]: status update time (8.252 seconds)
Dec 25 22:38:45 pveProliant pvestatd[1442]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Dec 25 22:38:46 pveProliant pvestatd[1442]: status update time (8.301 seconds)

 

[scroll_dp]

update:
After some time(work.....), I let it go by night and it takes so long (If I remember correctly, 1 and half year ago it takes only few minutes, just read to 100% progress, than after approximately 5 min it finished), about 1 hour for one VM, but it finished OK. But still, by this process, the whole server freezes by this operation, only web interface in half was working properly and remote ssh session work(delay for command). Nothing else.