Test migration stuck

[czechsys]

PVE remotes:

proxmox-ve: 8.4.0 (running kernel: 6.8.12-11-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)

PDM:
Installed: 0.1.11

VM 10 GB + 1 TB disks

All endpoints 3 devices are on the same subnet without fw, 10 Gbps network, hdd raid. This was 2nd try, 1st try was stuck on end of 1st disk.

From:

2025-07-23 12:08:58 remote: started tunnel worker 'UPID:pve-backup-2:002E85EE:16AA56A1:6880B4BA:qmtunnel:103:root@pam!pdm-admin:'
tunnel: -> sending command "version" to remote
tunnel: <- got reply
2025-07-23 12:09:03 local WS tunnel version: 2
2025-07-23 12:09:03 remote WS tunnel version: 2
2025-07-23 12:09:03 minimum required WS tunnel version: 2
websocket tunnel started
2025-07-23 12:09:03 starting migration of VM 101 to node 'pve-backup-2' (pve-backup-2)
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
2025-07-23 12:09:03 found local disk 'pve:vm-101-disk-0' (attached)
2025-07-23 12:09:03 found local disk 'pve:vm-101-disk-1' (attached)
2025-07-23 12:09:03 copying local disk images
tunnel: -> sending command "disk-import" to remote
tunnel: <- got reply
tunnel: accepted new connection on '/run/pve/101.storage'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/pve/101.storage'
262144 bytes (262 kB, 256 KiB) copied, 4 s, 64.7 kB/s327680 bytes (328 kB, 320 KiB) copied, 4 s, 80.9 kB/s393216 bytes (393 kB, 384 KiB) copied, 4 s, 97.0 kB/s458752 bytes (459 kB, 448 KiB) copied, 4 s, 113 kB/s 238747648 bytes (239 MB, 228 MiB) copied, 5 s, 47.7 MB/s478150656 bytes (478 MB, 456 MiB) copied, 6 s, 79.7 MB/s720896000 bytes (721 MB, 688 MiB) copied, 7 s, 103 MB/s 960888832 bytes (961 MB, 916 MiB) copied, 8 s, 120 MB/s1210712064 bytes (1.2 GB, 1.1 GiB) copied, 9 s, 135 MB/s1458307072 bytes (1.5 GB, 1.4 GiB) copied, 10 s, 146 MB/s1674313728 bytes (1.7 GB, 1.6 GiB) copied, 11 s, 152 MB/s1866399744 bytes (1.9 GB, 1.7 GiB) copied, 12 s, 156 MB/s2096889856 bytes (2.1 GB, 2.0 GiB) copied, 13 s, 161 MB/s2328625152 bytes (2.3 GB, 2.2 GiB) copied, 14 s, 166 MB/s2565341184 bytes (2.6 GB, 2.4 GiB) copied, 15 s, 171 MB/s2826960896 bytes (2.8 GB, 2.6 GiB) copied, 16 s, 177 MB/s3074228224 bytes (3.1 GB, 2.9 GiB) copied, 17 s, 181 MB/s3307012096 bytes (3.3 GB, 3.1 GiB) copied, 18 s, 184 MB/s3562209280 bytes (3.6 GB, 3.3 GiB) copied, 19 s, 187 MB/s3801415680 bytes (3.8 GB, 3.5 GiB) copied, 20 s, 190 MB/s4049272832 bytes (4.0 GB, 3.8 GiB) copied, 21 s, 193 MB/s4291362816 bytes (4.3 GB, 4.0 GiB) copied, 22 s, 195 MB/s4517068800 bytes (4.5 GB, 4.2 GiB) copied, 23 s, 196 MB/s4745658368 bytes (4.7 GB, 4.4 GiB) copied, 24 s, 198 MB/s4965531648 bytes (5.0 GB, 4.6 GiB) copied, 25 s, 199 MB/s5178654720 bytes (5.2 GB, 4.8 GiB) copied, 26 s, 199 MB/s5382078464 bytes (5.4 GB, 5.0 GiB) copied, 27 s, 199 MB/s5590810624 bytes (5.6 GB, 5.2 GiB) copied, 28 s, 200 MB/s5818286080 bytes (5.8 GB, 5.4 GiB) copied, 29 s, 201 MB/s6034292736 bytes (6.0 GB, 5.6 GiB) copied, 30 s, 201 MB/s6249775104 bytes (6.2 GB, 5.8 GiB) copied, 31 s, 202 MB/s6479282176 bytes (6.5 GB, 6.0 GiB) copied, 32 s, 202 MB/s6706888704 bytes (6.7 GB, 6.2 GiB) copied, 33 s, 203 MB/s6935085056 bytes (6.9 GB, 6.5 GiB) copied, 34 s, 204 MB/s7161970688 bytes (7.2 GB, 6.7 GiB) copied, 35 s, 205 MB/s7382564864 bytes (7.4 GB, 6.9 GiB) copied, 36 s, 205 MB/s7610236928 bytes (7.6 GB, 7.1 GiB) copied, 37 s, 206 MB/s7834959872 bytes (7.8 GB, 7.3 GiB) copied, 38 s, 206 MB/s8064729088 bytes (8.1 GB, 7.5 GiB) copied, 39 s, 207 MB/s8299216896 bytes (8.3 GB, 7.7 GiB) copied, 40 s, 207 MB/s8522825728 bytes (8.5 GB, 7.9 GiB) copied, 41 s, 208 MB/s8758493184 bytes (8.8 GB, 8.2 GiB) copied, 42 s, 209 MB/s8994488320 bytes (9.0 GB, 8.4 GiB) copied, 43 s, 209 MB/s9205579776 bytes (9.2 GB, 8.6 GiB) copied, 44 s, 209 MB/s9443475456 bytes (9.4 GB, 8.8 GiB) copied, 45 s, 210 MB/s9671540736 bytes (9.7 GB, 9.0 GiB) copied, 46 s, 210 MB/s9907929088 bytes (9.9 GB, 9.2 GiB) copied, 47 s, 211 MB/s10138222592 bytes (10 GB, 9.4 GiB) copied, 48 s, 211 MB/s10359406592 bytes (10 GB, 9.6 GiB) copied, 49 s, 211 MB/s10584129536 bytes (11 GB, 9.9 GiB) copied, 50 s, 212 MB/s
163840+0 records in
163840+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 51.6712 s, 208 MB/s
tunnel: -> sending command "query-disk-import" to remote
tunnel: done handling forwarded connection from '/run/pve/101.storage'
tunnel: <- got reply
2025-07-23 12:09:57 disk-import: volume datavg1/vm-103-disk-0 already exists - importing with a different name
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-07-23 12:09:58 disk-import:   Logical volume "vm-103-disk-2" created.
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-07-23 12:10:04 waiting for disk import to finish..
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-07-23 12:10:10 waiting for disk import to finish..
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-07-23 12:10:11 disk-import: 3232+639798 records in
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-07-23 12:10:12 disk-import: 3232+639798 records out
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-07-23 12:10:13 disk-import: 10737418240 bytes (11 GB, 10 GiB) copied, 60.4283 s, 178 MB/s
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-07-23 12:10:14 volume 'pve:vm-101-disk-0' is 'datavg1:vm-103-disk-2' on the target
tunnel: -> sending command "disk-import" to remote
tunnel: <- got reply
tunnel: accepted new connection on '/run/pve/101.storage'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/pve/101.storage'
262144 bytes (262 kB, 256 KiB) copied, 4 s, 65.4 kB/s327680 bytes (328 kB, 320 KiB) copied, 4 s, 81.8 kB/s393216 bytes (393 kB, 384 KiB) copied, 4 s, 98.1 kB/s458752 bytes (459 kB, 448 KiB) copied, 4 s, 114 kB/s

To:

mtunnel started
received command 'version'
received command 'bwlimit'
received command 'bwlimit'
received command 'disk-import'
ready

received command 'ticket'
received command 'query-disk-import'
disk-import: volume datavg1/vm-103-disk-0 already exists - importing with a different name

received command 'query-disk-import'
disk-import:   Logical volume "vm-103-disk-2" created.

received command 'query-disk-import'
received command 'query-disk-import'
received command 'query-disk-import'
disk-import: 3232+639798 records in

received command 'query-disk-import'
disk-import: 3232+639798 records out

received command 'query-disk-import'
disk-import: 10737418240 bytes (11 GB, 10 GiB) copied, 60.4283 s, 178 MB/s

received command 'query-disk-import'
disk-import: successfully imported 'datavg1:vm-103-disk-2'

received command 'disk-import'
ready

received command 'ticket'

PDM:

mtunnel startedreceived command 'version'received command 'bwlimit'received command 'bwlimit'received command 'disk-import'readyreceived command 'ticket'received command 'query-disk-import'disk-import: volume datavg1/vm-103-disk-0 already exists - importing with a different namereceived command 'query-disk-import'disk-import:   Logical volume "vm-103-disk-2" created.received command 'query-disk-import'received command 'query-disk-import'received command 'query-disk-import'disk-import: 3232+639798 records inreceived command 'query-disk-import'disk-import: 3232+639798 records outreceived command 'query-disk-import'disk-import: 10737418240 bytes (11 GB, 10 GiB) copied, 60.4283 s, 178 MB/sreceived command 'query-disk-import'disk-import: successfully imported 'datavg1:vm-103-disk-2'received command 'disk-import'readyreceived command 'ticket'

 

[candidnirav]

Thanks for sharing

 

[LukasInCloud]

That's a well-structured outreach approach. It's concise yet persuasive.

 

[fiona]

Hi,

That's a well-structured outreach approach. It's concise yet persuasive.

I'm sorry but what do you mean?

 

[LukasInCloud]

Hi,

I'm sorry but what do you mean?

I meant that the logs are well-structured and indicate that the target disk had I/O errors, so Proxmox automatically moved the VM image to a new volume.

 

[fiona]

I meant that the logs are well-structured and indicate that the target disk had I/O errors, so Proxmox automatically moved the VM image to a new volume.

The image was renamed during migration because there was already a left-over that had a conflicting name on the target side. Where do you see an indication of IO errors?

 

[czechsys]

The same test as with Alpha version. The same remotes, upgraded to PVE 9. Migration of the 2nd disk stucked, manually stopped.

Remotes: pve-manager/9.0.6/49c767b70aeb6648 (running kernel: 6.14.11-1-pve)

2025-09-19 11:28:53 remote: started tunnel worker 'UPID:SERVER2:000DF063:08BC0A75:68CD2255:qmtunnel:104:root@pam!pdm-admin:'
tunnel: -> sending command "version" to remote
tunnel: <- got reply
2025-09-19 11:28:53 local WS tunnel version: 2
2025-09-19 11:28:53 remote WS tunnel version: 2
2025-09-19 11:28:53 minimum required WS tunnel version: 2
websocket tunnel started
2025-09-19 11:28:53 starting migration of VM 104 to node 'SERVER2' (SERVER2_FQDN)
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
2025-09-19 11:28:53 found local disk 'pve:vm-104-disk-0' (attached)
2025-09-19 11:28:53 found local disk 'pve:vm-104-disk-1' (attached)
2025-09-19 11:28:53 copying local disk images
tunnel: -> sending command "disk-import" to remote
tunnel: <- got reply
tunnel: accepted new connection on '/run/pve/104.storage'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/pve/104.storage'

246087680 bytes (246 MB, 235 MiB) copied, 1 s, 244 MB/s
...
10736304128 bytes (11 GB, 10 GiB) copied, 134 s, 80.0 MB/s
163840+0 records in
163840+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 135.209 s, 79.4 MB/s
tunnel: -> sending command "query-disk-import" to remote
tunnel: done handling forwarded connection from '/run/pve/104.storage'
tunnel: <- got reply
2025-09-19 11:31:11 disk-import:   Wiping dos signature on /dev/datavg1/vm-104-disk-0.
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:12 disk-import:   Logical volume "vm-104-disk-0" created.
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:18 waiting for disk import to finish..
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:19 disk-import: 20907+563462 records in
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:20 disk-import: 20907+563462 records out
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:21 disk-import: 10737418240 bytes (11 GB, 10 GiB) copied, 143.124 s, 75.0 MB/s
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:22 volume 'pve:vm-104-disk-0' is 'datavg1:vm-104-disk-0' on the target
tunnel: -> sending command "disk-import" to remote
tunnel: <- got reply
tunnel: accepted new connection on '/run/pve/104.storage'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/pve/104.storage'

255131648 bytes (255 MB, 243 MiB) copied, 1 s, 255 MB/s
...
22566666240 bytes (23 GB, 21 GiB) copied, 103 s, 220 MB/s
...
22689808384 bytes (23 GB, 21 GiB) copied, 428 s, 53.1 MB/scommand 'set -o pipefail && pvesm export pve:vm-104-disk-1 raw+size - -with-snapshots 0' failed: interrupted by signal

CMD websocket tunnel died: command 'proxmox-websocket-tunnel' failed: interrupted by signal

Use of uninitialized value $res in concatenation (.) or string at /usr/share/perl5/PVE/Tunnel.pm line 120, <GEN1342> line 13.
2025-09-19 11:38:35 ERROR: failed to decode tunnel reply '' (command '{"cmd":"query-disk-import"}') - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/PVE/Tunnel.pm line 119, <GEN1342> line 13.
2025-09-19 11:38:35 aborting phase 1 - cleanup resources
2025-09-19 11:38:36 ERROR: writing to tunnel failed: broken pipe
2025-09-19 11:38:36 ERROR: migration aborted (duration 00:09:43): failed to decode tunnel reply '' (command '{"cmd":"query-disk-import"}') - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/PVE/Tunnel.pm line 119, <GEN1342> line 13.
TASK ERROR: migration aborted

2025-09-19 11:28:53 remote: started tunnel worker 'UPID:SERVER2:000DF063:08BC0A75:68CD2255:qmtunnel:104:root@pam!pdm-admin:'
tunnel: -> sending command "version" to remote
tunnel: <- got reply
2025-09-19 11:28:53 local WS tunnel version: 2
2025-09-19 11:28:53 remote WS tunnel version: 2
2025-09-19 11:28:53 minimum required WS tunnel version: 2
websocket tunnel started
2025-09-19 11:28:53 starting migration of VM 104 to node 'SERVER2' (SERVER2_FQDN)
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
2025-09-19 11:28:53 found local disk 'pve:vm-104-disk-0' (attached)
2025-09-19 11:28:53 found local disk 'pve:vm-104-disk-1' (attached)
2025-09-19 11:28:53 copying local disk images
tunnel: -> sending command "disk-import" to remote
tunnel: <- got reply
tunnel: accepted new connection on '/run/pve/104.storage'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/pve/104.storage'

246087680 bytes (246 MB, 235 MiB) copied, 1 s, 244 MB/s
...
10736304128 bytes (11 GB, 10 GiB) copied, 134 s, 80.0 MB/s
163840+0 records in
163840+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 135.209 s, 79.4 MB/s
tunnel: -> sending command "query-disk-import" to remote
tunnel: done handling forwarded connection from '/run/pve/104.storage'
tunnel: <- got reply
2025-09-19 11:31:11 disk-import:   Wiping dos signature on /dev/datavg1/vm-104-disk-0.
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:12 disk-import:   Logical volume "vm-104-disk-0" created.
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:18 waiting for disk import to finish..
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:19 disk-import: 20907+563462 records in
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:20 disk-import: 20907+563462 records out
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:21 disk-import: 10737418240 bytes (11 GB, 10 GiB) copied, 143.124 s, 75.0 MB/s
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-09-19 11:31:22 volume 'pve:vm-104-disk-0' is 'datavg1:vm-104-disk-0' on the target
tunnel: -> sending command "disk-import" to remote
tunnel: <- got reply
tunnel: accepted new connection on '/run/pve/104.storage'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/pve/104.storage'

255131648 bytes (255 MB, 243 MiB) copied, 1 s, 255 MB/s
...
22566731776 bytes (23 GB, 21 GiB) copied, 103 s, 220 MB/s
...
22689808384 bytes (23 GB, 21 GiB) copied, 428 s, 53.1 MB/scommand 'set -o pipefail && pvesm export pve:vm-104-disk-1 raw+size - -with-snapshots 0' failed: interrupted by signal

CMD websocket tunnel died: command 'proxmox-websocket-tunnel' failed: interrupted by signal

Use of uninitialized value $res in concatenation (.) or string at /usr/share/perl5/PVE/Tunnel.pm line 120, <GEN1342> line 13.
2025-09-19 11:38:35 ERROR: failed to decode tunnel reply '' (command '{"cmd":"query-disk-import"}') - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/PVE/Tunnel.pm line 119, <GEN1342> line 13.
2025-09-19 11:38:35 aborting phase 1 - cleanup resources
2025-09-19 11:38:36 ERROR: writing to tunnel failed: broken pipe
2025-09-19 11:38:36 ERROR: migration aborted (duration 00:09:43): failed to decode tunnel reply '' (command '{"cmd":"query-disk-import"}') - malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/PVE/Tunnel.pm line 119, <GEN1342> line 13.
TASK ERROR: migration aborted

mtunnel started
received command 'version'
received command 'bwlimit'
received command 'bwlimit'
received command 'disk-import'
ready

received command 'ticket'
received command 'query-disk-import'
disk-import:   Wiping dos signature on /dev/datavg1/vm-104-disk-0.

received command 'query-disk-import'
disk-import:   Logical volume "vm-104-disk-0" created.

received command 'query-disk-import'
received command 'query-disk-import'
disk-import: 20907+563462 records in

received command 'query-disk-import'
disk-import: 20907+563462 records out

received command 'query-disk-import'
disk-import: 10737418240 bytes (11 GB, 10 GiB) copied, 143.124 s, 75.0 MB/s

received command 'query-disk-import'
disk-import: successfully imported 'datavg1:vm-104-disk-0'

received command 'disk-import'
ready

received command 'ticket'
TASK ERROR: mtunnel exited unexpectedly

 

[fiona]

Hi,
what storage types are pve and datavg1? I assume the latter is LVM, but think or thin? Do you see any load, be it network or CPU or IO caused by the task on the source or target node while it is "stuck"? You could also do a ps faxl to what sub-process it's currently executing.

 

[czechsys]

Both sides are LVM thick (HP G8 servers with Pxxx smartaray raid10)

src:

4     0    1573       1  20   0 179628 137020 hrtime Ss  ?        272:12 pvestatd
0     0    1593       1  20   0 219244 175984 hrtime Ss  ?          0:31 pvedaemon
1     0   15971    1593  20   0 265144 196768 do_epo S   ?          0:33  \_ pvedaemon worker
1     0  916994    1593  20   0 266524 196768 do_epo S   ?          0:02  \_ pvedaemon worker
1     0  937248    1593  20   0 230632 161000 do_epo S   ?          0:01  \_ pvedaemon worker
5     0  960340  937248  20   0 238028 157536 do_sel Ss  ?          0:00      \_ task UPID:SERVER1:000EA754:08C70F03:68CD41EF:qmigrate:104:root@pam!pdm-admin:
1     0  960341  960340  20   0 237880 150664 do_sel S   ?          0:00          \_ task UPID:SERVER1:000EA754:08C70F03:68CD41EF:qmigrate:104:root@pam!pdm-admin:
0     0  960342  960341  20   0 1775560 13248 futex_ Sl  ?          1:08          |   \_ proxmox-websocket-tunnel
0     0  960812  960340  20   0 150060 104288 do_sel S   ?          0:00          \_ /usr/bin/perl /sbin/pvesm export pve:vm-104-disk-1 raw+size - -with-snapshots 0
0     0  960817  960812  20   0   2620  1620 sock_a S    ?          0:11              \_ dd if=/dev/pve/vm-104-disk-1 bs=64k status=progress

Strace for pvesm export:

strace: Process 960812 attached
pselect6(16, [8], NULL, NULL, {tv_sec=0, tv_nsec=245088769}, NULL) = 1 (in [8], left {tv_sec=0, tv_nsec=210124433})
read(8, "\r11926765568 bytes (12 GB, 11 Gi"..., 4096) = 118
write(2, "\r11926765568 bytes (12 GB, 11 Gi"..., 118) = 118
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 1 (in [8], left {tv_sec=0, tv_nsec=544421353})
read(8, "\r11928076288 bytes (12 GB, 11 Gi"..., 4096) = 118
write(2, "\r11928076288 bytes (12 GB, 11 Gi"..., 118) = 118
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 1 (in [8], left {tv_sec=0, tv_nsec=263064593})
read(8, "\r11929583616 bytes (12 GB, 11 Gi"..., 4096) = 59
write(2, "\r11929583616 bytes (12 GB, 11 Gi"..., 59) = 59
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 1 (in [8], left {tv_sec=0, tv_nsec=260078304})
read(8, "\r11931090944 bytes (12 GB, 11 Gi"..., 4096) = 118
write(2, "\r11931090944 bytes (12 GB, 11 Gi"..., 118) = 118
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 1 (in [8], left {tv_sec=0, tv_nsec=107720881})
read(8, "\r11932532736 bytes (12 GB, 11 Gi"..., 4096) = 59
write(2, "\r11932532736 bytes (12 GB, 11 Gi"..., 59) = 59
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 1 (in [8], left {tv_sec=0, tv_nsec=999329102})
read(8, "\r11932598272 bytes (12 GB, 11 Gi"..., 4096) = 59
write(2, "\r11932598272 bytes (12 GB, 11 Gi"..., 59) = 59
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 1 (in [8], left {tv_sec=0, tv_nsec=261038326})
read(8, "\r11934236672 bytes (12 GB, 11 Gi"..., 4096) = 118
write(2, "\r11934236672 bytes (12 GB, 11 Gi"..., 118) = 118
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL^Cstrace: Process 960812 detached
 <detached ...>

Strace for dd:

strace: Process 960817 attached
write(1, "REDACTED\0\310\0\0\0\0\0\0\0"..., 28992) = 28992
write(2, "\r", 1)                       = 1
write(2, "11986665472 bytes (12 GB, 11 GiB"..., 58) = 58
read(0, "\1\0\0\0\10\354\212\373\0\0\0\0\370\0X\1\0 \4 \0\0\0\0x\237\10\1\360\236\10\1"..., 65536) = 65536
write(1, "\1\0\0\0\10\354\212\373\0\0\0\0\370\0X\1\0 \4 \0\0\0\0x\237\10\1\360\236\10\1"..., 65536) = 65536
write(2, "\r", 1)                       = 1
write(2, "11986731008 bytes (12 GB, 11 GiB"..., 58) = 58
read(0, "\1\0\0\0\300\303\213\373\0\0\0\0\360\0\360\0\0 \4 \0\0\0\0p\237\30\1\330\236(\1"..., 65536) = 65536
write(1, "\1\0\0\0\300\303\213\373\0\0\0\0\360\0\360\0\0 \4 \0\0\0\0p\237\30\1\330\236(\1"..., 65536) = 65536
read(0, "\1\0\0\0 \235\214\373\0\0\0\0\354\0\0\1\0 \4 \0\0\0\0h\237(\1\320\236(\1"..., 65536) = 65536
write(1, "\1\0\0\0 \235\214\373\0\0\0\0\354\0\0\1\0 \4 \0\0\0\0h\237(\1\320\236(\1"..., 65536) = 65536
read(0, "\1\0\0\0\300v\215\373\0\0\0\0\354\0 \1\0 \4 \0\0\0\0h\237(\1\340\236\10\1"..., 65536) = 65536
write(1, "\1\0\0\0\300v\215\373\0\0\0\0\354\0 \1\0 \4 \0\0\0\0h\237(\1\340\236\10\1"..., 65536) = 65536
read(0, "\1\0\0\0pP\216\373\0\0\0\0\374\08\1\0 \4 \0\0\0\0X\237H\1\270\2368\1"..., 65536) = 65536
write(1, "\1\0\0\0pP\216\373\0\0\0\0\374\08\1\0 \4 \0\0\0\0X\237H\1\270\2368\1"..., 65536) = 65536
read(0, "\1\0\0\0p)\217\373\0\0\0\0\370\0H\1\0 \4 \0\0\0\0x\237\10\1\370\236\370\0"..., 65536) = 65536
write(1, "\1\0\0\0p)\217\373\0\0\0\0\370\0H\1\0 \4 \0\0\0\0x\237\10\1\370\236\370\0"..., 65536) = 65536
read(0, "\1\0\0\0`\4\220\373\0\0\0\0\340\0\350\0\0 \4 \0\0\0\0X\237H\1\270\2368\1"..., 65536) = 65536
write(1, "\1\0\0\0`\4\220\373\0\0\0\0\340\0\350\0\0 \4 \0\0\0\0X\237H\1\270\2368\1"..., 65536) = 65536
read(0, "\1\0\0\0\270\334\220\373\0\0\0\0\370\0\10\1\0 \4 \0\0\0\0x\237\10\1\350\236\30\1"..., 65536) = 65536
write(1, "\1\0\0\0\270\334\220\373\0\0\0\0\370\0\10\1\0 \4 \0\0\0\0x\237\10\1\350\236\30\1"..., 65536) = 65536
read(0, "\1\0\0\0\270\262\221\373\0\0\0\0\370\0@\1\0 \4 \0\0\0\0p\237\30\1\340\236\30\1"..., 65536) = 65536
write(1, "\1\0\0\0\270\262\221\373\0\0\0\0\370\0@\1\0 \4 \0\0\0\0p\237\30\1\340\236\30\1"..., 65536) = 65536
read(0, "\1\0\0\0\210\207\222\373\0\0\0\0\370\0`\1\0 \4 \0\0\0\0p\237\30\1\350\236\10\1"..., 65536) = 65536
write(1, "\1\0\0\0\210\207\222\373\0\0\0\0\370\0`\1\0 \4 \0\0\0\0p\237\30\1\350\236\10\1"..., 65536) = 65536
read(0, "\1\0\0\0000]\223\373\0\0\0\0\374\0\10\1\0 \4 \0\0\0\0p\237\30\1\350\236\10\1"..., 65536) = 65536
write(1, "\1\0\0\0000]\223\373\0\0\0\0\374\0\10\1\0 \4 \0\0\0\0p\237\30\1\350\236\10\1"..., 65536) = 65536
read(0, "\1\0\0\0\0103\224\373\0\0\0\0\374\0\30\1\0 \4 \0\0\0\0x\237\10\1\360\236\10\1"..., 65536) = 65536
write(1, "\1\0\0\0\0103\224\373\0\0\0\0\374\0\30\1\0 \4 \0\0\0\0x\237\10\1\360\236\10\1"..., 65536) = 65536
read(0, "\1\0\0\0@\t\225\373\0\0\0\0\374\0\30\1\0 \4 \0\0\0\0x\237\10\1\350\236\30\1"..., 65536) = 65536
write(1, "\1\0\0\0@\t\225\373\0\0\0\0\374\0\30\1\0 \4 \0\0\0\0x\237\10\1\350\236\30\1"..., 65536) = 65536
read(0, "\1\0\0\0\200\340\225\373\0\0\0\0\354\08\1\0 \4 \0\0\0\0`\2378\1\310\236(\1"..., 65536) = 65536
write(1, "\1\0\0\0\200\340\225\373\0\0\0\0\354\08\1\0 \4 \0\0\0\0`\2378\1\310\236(\1"..., 65536) = 65536
read(0, "\1\0\0\0\330\270\226\373\0\0\0\0\360\08\1\0 \4 \0\0\0\0p\237\30\1\350\236\20\1"..., 65536) = 65536
write(1, "\1\0\0\0\330\270\226\373\0\0\0\0\360\08\1\0 \4 \0\0\0\0p\237\30\1\350\236\20\1"..., 65536) = 65536
read(0, "\1\0\0\0X\222\227\373\0\0\0\0\370\0P\1\0 \4 \0\0\0\0`\2378\1\350\236\360\0"..., 65536) = 65536
write(1, "\1\0\0\0X\222\227\373\0\0\0\0\370\0P\1\0 \4 \0\0\0\0`\2378\1\350\236\360\0"..., 65536) = 65536
read(0, "\1\0\0\0\370l\230\373\0\0\0\0\364\0P\1\0 \4 \0\0\0\0x\237\10\1\330\2368\1"..., 65536) = 65536
write(1, "\1\0\0\0\370l\230\373\0\0\0\0\364\0P\1\0 \4 \0\0\0\0x\237\10\1\330\2368\1"..., 65536) = 65536
read(0, "\1\0\0\0\250F\231\373\0\0\0\0\354\0\0\1\0 \4 \0\0\0\0`\2378\1\300\2368\1"..., 65536) = 65536
write(1, "\1\0\0\0\250F\231\373\0\0\0\0\354\0\0\1\0 \4 \0\0\0\0`\2378\1\300\2368\1"..., 65536) = 65536
read(0, "\1\0\0\0x!\232\373\0\0\0\0\344\0P\1\0 \4 \0\0\0\0X\237H\1\260\236H\1"..., 65536) = 65536
write(1, "\1\0\0\0x!\232\373\0\0\0\0\344\0P\1\0 \4 \0\0\0\0X\237H\1\260\236H\1"..., 65536) = 65536
read(0, "\1\0\0\0\350\370\232\373\0\0\0\0\360\0P\1\0 \4 \0\0\0\0`\2378\1\330\236\10\1"..., 65536) = 65536
write(1, "\1\0\0\0\350\370\232\373\0\0\0\0\360\0P\1\0 \4 \0\0\0\0`\2378\1\330\236\10\1"..., 65536) = 65536
read(0, "\1\0\0\0(\321\233\373\0\0\0\0\354\08\1\0 \4 \0\0\0\0`\2378\1\320\236\30\1"..., 65536) = 65536
write(1, "\1\0\0\0(\321\233\373\0\0\0\0\354\08\1\0 \4 \0\0\0\0`\2378\1\320\236\30\1"..., 65536) = 65536
read(0, "\1\0\0\0\20\250\234\373\0\0\0\0\364\0\30\1\0 \4 \0\0\0\0x\237\10\1\360\236\10\1"..., 65536) = 65536
write(1, "\1\0\0\0\20\250\234\373\0\0\0\0\364\0\30\1\0 \4 \0\0\0\0x\237\10\1\360\236\10\1"..., 65536) = 65536
read(0, "\1\0\0\0\260\200\235\373\0\0\0\0\360\0(\1\0 \4 \0\0\0\0`\2378\1\300\2368\1"..., 65536) = 65536
write(1, "\1\0\0\0\260\200\235\373\0\0\0\0\360\0(\1\0 \4 \0\0\0\0`\2378\1\300\2368\1"..., 65536) = 65536
write(2, "\r", 1)                       = 1
write(2, "11988172800 bytes (12 GB, 11 GiB"..., 58) = 58

target:

1     0   17763       1  20   0 218764 146980 hrtime Ss  ?          0:23 pvedaemon
1     0   17764   17763  20   0 247640 175232 do_sel S   ?          0:49  \_ pvedaemon worker
5     0  965410   17764  20   0 253844 170944 unix_s Ss  ?          0:00  |   \_ task UPID:SERVER2:000EBB22:08C86288:68CD41EF:qmtunnel:104:root@pam!pdm-admin:
0     0  965817  965410  20   0 155276 103844 do_sel S   ?          0:00  |       \_ /usr/bin/perl /sbin/pvesm import datavg1:vm-104-disk-1 raw+size unix:///run/qemu-server/104.storage
0     0  965840  965817  20   0   2620  1428 unix_s S    ?          0:09  |           \_ dd of=/dev/datavg1/vm-104-disk-1 bs=64k
1     0   17765   17763  20   0 243100 171648 do_sel S   ?          0:49  \_ pvedaemon worker
1     0   17766   17763  20   0 234700 162932 do_sel S   ?          0:43  \_ pvedaemon worker

Strace for pvesm import:

strace: Process 965817 attached
pselect6(16, [8], NULL, NULL, {tv_sec=0, tv_nsec=854123427}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL) = 0 (Timeout)
pselect6(16, [8], NULL, NULL, {tv_sec=1, tv_nsec=0}, NULL^Cstrace: Process 965817 detached
 <detached ...>

Strace for dd:

strace: Process 965840 attached
read(0, "]:25\0\0\0\0\223\2\0\0\0\0\0\0\7\2\0\0\0\0\0\0\0\0\0\0\17\0d\256"..., 65536) = 65536
write(1, "]:25\0\0\0\0\223\2\0\0\0\0\0\0\7\2\0\0\0\0\0\0\0\0\0\0\17\0d\256"..., 65536) = 65536
read(0, "6]:25\0\0\0\223\2\0\0\0\0\0\0\7\2\0\0\0\0\0\0\0\0\0\0\17\0l\256"..., 65536) = 65536
write(1, "6]:25\0\0\0\223\2\0\0\0\0\0\0\7\2\0\0\0\0\0\0\0\0\0\0\17\0l\256"..., 65536) = 65536
read(0, ".google.com[74.125.136.26]:25\0\0\0"..., 65536) = 65536
write(1, ".google.com[74.125.136.26]:25\0\0\0"..., 65536) = 65536
read(0, "5\0\0\0\223\2\0\0\7\2\0\0\0\0\0\0\0\0\0\0\17\0|\256*\0\f\0\3\t \177"..., 65536) = 59200
write(1, "5\0\0\0\223\2\0\0\7\2\0\0\0\0\0\0\0\0\0\0\17\0|\256*\0\f\0\3\t \177"..., 59200) = 59200
read(0, "\300\340\332ua\311\1\0[gmail-smtp-in.l.google."..., 65536) = 65536
write(1, "\300\340\332ua\311\1\0[gmail-smtp-in.l.google."..., 65536) = 65536
<detached ...>

It looks as it's crawled to old turtle's walk, almost 0% iowait/softirq.

Fri Sep 19 02:09:37 PM CEST 2025
write(2, "12535070720 bytes (13 GB, 12 GiB"..., 58) = 58
write(2, "12543918080 bytes (13 GB, 12 GiB"..., 58) = 58
Fri Sep 19 02:09:50 PM CEST 2025

 

[fiona]

Oh, so it's never fully stuck, just extremely slow? To find out whether the issue originates on the source or on the target, you could try on the source, using dd if=/dev/pve/vm-104-disk-1 bs=64k status=progress of=/dev/null and similarly, allocating the image manually on the target and then using dd of=/dev/datavg1/testimage bs=64k status=progress if=/dev/urandom. This should tell you whether it's reading or writing. If it's neither, it might be the interaction with the network or at least triggered by that.