PVE 6.3.6 - Live migration failed after two attempts: qmp command 'query-migrate' failed - client closed connection

[mpizzolo]

drive-scsi2: transferred: 298232446976 bytes remaining: 0 bytes total: 298232446976 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
2021-04-20 14:07:58 volume 'disk2b_encrypted:vm-521-disk-0' is 'node04_disk10:vm-521-disk-0' on the target
2021-04-20 14:07:58 starting online/live migration on tcp:10.10.1.27:60000
2021-04-20 14:07:58 set migration_caps
2021-04-20 14:07:58 migration speed limit: 20971520000 B/s
2021-04-20 14:07:58 migration downtime limit: 100 ms
2021-04-20 14:07:58 migration cachesize: 2147483648 B
2021-04-20 14:07:58 set migration parameters
2021-04-20 14:07:58 start migrate command to tcp:10.10.1.27:60000
2021-04-20 14:07:59 migration status: active (transferred 1095561559, remaining 11526504448), total 12902572032)
2021-04-20 14:07:59 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:00 migration status: active (transferred 2272136294, remaining 10294976512), total 12902572032)
2021-04-20 14:08:00 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:01 migration status: active (transferred 3447517819, remaining 9084678144), total 12902572032)
2021-04-20 14:08:01 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:02 migration status: active (transferred 4623724632, remaining 7899500544), total 12902572032)
2021-04-20 14:08:02 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:03 migration status: active (transferred 5801991043, remaining 6721835008), total 12902572032)
2021-04-20 14:08:03 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:04 migration status: active (transferred 6979233570, remaining 5544235008), total 12902572032)
2021-04-20 14:08:04 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:05 migration status: active (transferred 8156463517, remaining 4364881920), total 12902572032)
2021-04-20 14:08:05 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:06 migration status: active (transferred 9332546764, remaining 3178168320), total 12902572032)
2021-04-20 14:08:06 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:07 migration status: active (transferred 10506745933, remaining 1997189120), total 12902572032)
2021-04-20 14:08:07 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:08 migration status: active (transferred 11673994860, remaining 738406400), total 12902572032)
2021-04-20 14:08:08 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:08 migration status: active (transferred 11795693443, remaining 615739392), total 12902572032)
2021-04-20 14:08:08 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:09 migration status: active (transferred 11916621871, remaining 493211648), total 12902572032)
2021-04-20 14:08:09 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:09 migration status: active (transferred 12038377964, remaining 370462720), total 12902572032)
2021-04-20 14:08:09 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:09 migration status: active (transferred 12156930642, remaining 251953152), total 12902572032)
2021-04-20 14:08:09 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:09 migration status: active (transferred 12278865052, remaining 130052096), total 12902572032)
2021-04-20 14:08:09 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 0 overflow 0
2021-04-20 14:08:09 migration status: active (transferred 12391385921, remaining 244465664), total 12902572032)
2021-04-20 14:08:09 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 344 overflow 0
2021-04-20 14:08:09 migration status: active (transferred 12441760345, remaining 167235584), total 12902572032)
2021-04-20 14:08:09 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 12603 overflow 0
2021-04-20 14:08:09 migration status: active (transferred 12492787640, remaining 98566144), total 12902572032)
2021-04-20 14:08:09 migration xbzrle cachesize: 2147483648 transferred 0 pages 0 cachemiss 25029 overflow 0
query migrate failed: VM 521 qmp command 'query-migrate' failed - client closed connection

2021-04-20 14:08:09 query migrate failed: VM 521 qmp command 'query-migrate' failed - client closed connection
query migrate failed: VM 521 not running

2021-04-20 14:08:10 query migrate failed: VM 521 not running
query migrate failed: VM 521 not running

2021-04-20 14:08:12 query migrate failed: VM 521 not running
query migrate failed: VM 521 not running

2021-04-20 14:08:13 query migrate failed: VM 521 not running
query migrate failed: VM 521 not running

2021-04-20 14:08:14 query migrate failed: VM 521 not running
query migrate failed: VM 521 not running

2021-04-20 14:08:15 query migrate failed: VM 521 not running
2021-04-20 14:08:15 ERROR: online migrate failure - too many query migrate failures - aborting
2021-04-20 14:08:15 aborting phase 2 - cleanup resources
2021-04-20 14:08:15 migrate_cancel
2021-04-20 14:08:15 migrate_cancel error: VM 521 not running
drive-scsi1: Cancelling block job
drive-scsi2: Cancelling block job
2021-04-20 14:08:15 ERROR: VM 521 not running
2021-04-20 14:08:22 ERROR: migration finished with problems (duration 00:28:36)
TASK ERROR: migration problems

 

[Stefan_R]

Please also post any logs from around the time this happened (journalctl -e from both source and destination node). Also, it would be helpful to describe the actual setup, post any VM configs, more info about what you tried to do, reproducability, etc... to help diagnose the issue. Just the task log is not enough unfortunately, as this can have a variety of underlying reasons.