qm remote-migrate ERROR

jmjosebest

Renowned Member
Jan 16, 2009
191
24
83
Hello, I've tried a remote migration of a 5 TB VM, and after 5 hours I got this error :(
Any idea what is the problem?
Thanks!

Code:
root@ns1000:~# qm remote-migrate 3107 3107 'apitoken=PVEAPIToken=root@pam!xdxd=bla-bla,host=ns1001' --target-bridge vmbr0 --target-storage backup --online 1
...
drive-scsi1: transferred 5.1 TiB of 5.1 TiB (100.00%) in 5h 8m 30s
drive-scsi1: transferred 5.1 TiB of 5.1 TiB (100.00%) in 5h 8m 31s
drive-scsi1: transferred 5.1 TiB of 5.1 TiB (100.00%) in 5h 8m 32s
drive-scsi1: transferred 5.1 TiB of 5.1 TiB (100.00%) in 5h 8m 33s
drive-scsi1: transferred 5.1 TiB of 5.1 TiB (100.00%) in 5h 8m 35s
drive-scsi1: transferred 5.1 TiB of 5.1 TiB (100.00%) in 5h 8m 36s
drive-scsi1: transferred 5.1 TiB of 5.1 TiB (100.00%) in 5h 8m 37s, ready
all 'mirror' jobs are ready
2023-09-17 19:28:50 starting online/live migration on unix:/run/qemu-server/3107.migrate
2023-09-17 19:28:50 set migration capabilities
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
2023-09-17 19:28:50 migration downtime limit: 100 ms
2023-09-17 19:28:50 migration cachesize: 4.0 GiB
2023-09-17 19:28:50 set migration parameters
2023-09-17 19:28:50 start migrate command to unix:/run/qemu-server/3107.migrate
tunnel: accepted new connection on '/run/qemu-server/3107.migrate'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/qemu-server/3107.migrate'
2023-09-17 19:28:51 migration active, transferred 320.5 MiB of 24.0 GiB VM-state, 331.8 MiB/s
2023-09-17 19:28:52 migration active, transferred 630.2 MiB of 24.0 GiB VM-state, 318.9 MiB/s
2023-09-17 19:28:53 migration active, transferred 958.7 MiB of 24.0 GiB VM-state, 331.8 MiB/s
2023-09-17 19:28:54 migration active, transferred 1.3 GiB of 24.0 GiB VM-state, 335.0 MiB/s
...
2023-09-17 19:30:15 migration active, transferred 27.3 GiB of 24.0 GiB VM-state, 338.9 MiB/s
2023-09-17 19:30:16 migration active, transferred 27.6 GiB of 24.0 GiB VM-state, 348.1 MiB/s
2023-09-17 19:30:17 migration active, transferred 28.0 GiB of 24.0 GiB VM-state, 331.8 MiB/s
2023-09-17 19:30:18 migration active, transferred 28.3 GiB of 24.0 GiB VM-state, 1.1 GiB/s
2023-09-17 19:30:18 xbzrle: send updates to 84703 pages in 13.9 MiB encoded memory, cache-miss 97.59%, overflow 39
2023-09-17 19:30:19 migration active, transferred 28.7 GiB of 24.0 GiB VM-state, 482.5 MiB/s
2023-09-17 19:30:19 xbzrle: send updates to 139744 pages in 24.3 MiB encoded memory, cache-miss 97.59%, overflow 641
2023-09-17 19:30:20 migration active, transferred 29.0 GiB of 24.0 GiB VM-state, 422.1 MiB/s
2023-09-17 19:30:20 xbzrle: send updates to 169985 pages in 30.1 MiB encoded memory, cache-miss 97.59%, overflow 923
2023-09-17 19:30:21 migration active, transferred 29.3 GiB of 24.0 GiB VM-state, 369.8 MiB/s
2023-09-17 19:30:21 xbzrle: send updates to 185949 pages in 32.8 MiB encoded memory, cache-miss 97.59%, overflow 986
2023-09-17 19:30:22 migration active, transferred 29.7 GiB of 24.0 GiB VM-state, 337.1 MiB/s
2023-09-17 19:30:22 xbzrle: send updates to 190283 pages in 33.6 MiB encoded memory, cache-miss 97.59%, overflow 1004
2023-09-17 19:30:24 migration active, transferred 30.0 GiB of 24.0 GiB VM-state, 477.0 MiB/s
2023-09-17 19:30:24 xbzrle: send updates to 209071 pages in 34.6 MiB encoded memory, cache-miss 69.44%, overflow 1011
2023-09-17 19:30:25 migration active, transferred 30.3 GiB of 24.0 GiB VM-state, 501.2 MiB/s
2023-09-17 19:30:25 xbzrle: send updates to 315423 pages in 50.4 MiB encoded memory, cache-miss 69.44%, overflow 1208
2023-09-17 19:30:26 migration active, transferred 30.7 GiB of 24.0 GiB VM-state, 430.7 MiB/s
2023-09-17 19:30:26 xbzrle: send updates to 347499 pages in 58.8 MiB encoded memory, cache-miss 69.44%, overflow 1406
2023-09-17 19:30:27 migration active, transferred 31.0 GiB of 24.0 GiB VM-state, 429.6 MiB/s
2023-09-17 19:30:27 xbzrle: send updates to 364185 pages in 62.3 MiB encoded memory, cache-miss 69.44%, overflow 1557
2023-09-17 19:30:28 migration active, transferred 31.4 GiB of 24.0 GiB VM-state, 595.0 MiB/s
2023-09-17 19:30:28 xbzrle: send updates to 481812 pages in 81.7 MiB encoded memory, cache-miss 63.94%, overflow 2690
2023-09-17 19:30:29 migration active, transferred 31.7 GiB of 24.0 GiB VM-state, 611.6 MiB/s
2023-09-17 19:30:29 xbzrle: send updates to 523963 pages in 94.5 MiB encoded memory, cache-miss 63.94%, overflow 4728
2023-09-17 19:30:30 auto-increased downtime to continue migration: 200 ms
2023-09-17 19:30:30 migration active, transferred 32.0 GiB of 24.0 GiB VM-state, 688.9 MiB/s
2023-09-17 19:30:30 xbzrle: send updates to 679815 pages in 119.9 MiB encoded memory, cache-miss 51.18%, overflow 7912
2023-09-17 19:30:31 migration active, transferred 32.4 GiB of 24.0 GiB VM-state, 746.2 MiB/s
2023-09-17 19:30:31 xbzrle: send updates to 846966 pages in 150.1 MiB encoded memory, cache-miss 40.50%, overflow 10548
2023-09-17 19:30:31 auto-increased downtime to continue migration: 400 ms
2023-09-17 19:30:32 migration active, transferred 32.7 GiB of 24.0 GiB VM-state, 1.5 GiB/s
2023-09-17 19:30:32 xbzrle: send updates to 1042766 pages in 177.5 MiB encoded memory, cache-miss 33.80%, overflow 12187
tunnel: done handling forwarded connection from '/run/qemu-server/3107.migrate'
tunnel: done handling forwarded connection from '/run/qemu-server/3107_nbd.migrate'
tunnel: done handling forwarded connection from '/run/qemu-server/3107_nbd.migrate'
query migrate failed: VM 3107 qmp command 'query-migrate' failed - client closed connection

2023-09-17 19:30:33 query migrate failed: VM 3107 qmp command 'query-migrate' failed - client closed connection
query migrate failed: VM 3107 not running

2023-09-17 19:30:34 query migrate failed: VM 3107 not running
query migrate failed: VM 3107 not running

2023-09-17 19:30:35 query migrate failed: VM 3107 not running
query migrate failed: VM 3107 not running

2023-09-17 19:30:36 query migrate failed: VM 3107 not running
query migrate failed: VM 3107 not running

2023-09-17 19:30:37 query migrate failed: VM 3107 not running
query migrate failed: VM 3107 not running

2023-09-17 19:30:38 query migrate failed: VM 3107 not running
2023-09-17 19:30:38 ERROR: online migrate failure - too many query migrate failures - aborting
2023-09-17 19:30:38 aborting phase 2 - cleanup resources
2023-09-17 19:30:38 migrate_cancel
2023-09-17 19:30:38 migrate_cancel error: VM 3107 not running
2023-09-17 19:30:38 ERROR: query-status error: VM 3107 not running
drive-scsi1: Cancelling block job
drive-scsi0: Cancelling block job
2023-09-17 19:30:38 ERROR: VM 3107 not running
tunnel: -> sending command "stop" to remote
tunnel: <- got reply
tunnel: -> sending command "quit" to remote
tunnel: <- got reply
2023-09-17 19:30:40 ERROR: migration finished with problems (duration 05:13:23)
migration problems
root@ns1000:~#
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!