Live migration failed for seemingly no reason

[runbgp]

I was migrating 3 VMs off a host in order to update and reboot it, and the final (and largest) VM failed and was left in a powered-off state. This was a migration from local storage to local storage on another, identical host. What went wrong, and why? Let me know if I can provide any further information. Thanks!

Proxmox VE 7.1-10
Linux 5.13.19-3-pve #1 SMP PVE 5.13.19-7 (Thu, 20 Jan 2022 16:37:56 +0100)
pve-manager/7.1-10/6ddebafe
2x Intel Xeon 2680v2
128GB RAM
Intel X520-DA2 in a 2x10G LAG
LSI 3108 w/ 6 SSDs in RAID10

drive-virtio0: transferred 99.5 GiB of 100.0 GiB (99.41%) in 4m 8s
drive-virtio0: transferred 99.9 GiB of 100.0 GiB (99.81%) in 4m 9s
drive-virtio0: transferred 100.0 GiB of 100.0 GiB (100.00%) in 4m 10s, ready
all 'mirror' jobs are ready
2022-03-06 19:44:30 starting online/live migration on unix:/run/qemu-server/107.migrate
2022-03-06 19:44:30 set migration capabilities
2022-03-06 19:44:30 migration downtime limit: 100 ms
2022-03-06 19:44:30 migration cachesize: 512.0 MiB
2022-03-06 19:44:30 set migration parameters
2022-03-06 19:44:30 start migrate command to unix:/run/qemu-server/107.migrate
2022-03-06 19:44:31 migration active, transferred 406.4 MiB of 4.0 GiB VM-state, 424.8 MiB/s
2022-03-06 19:44:32 migration active, transferred 818.1 MiB of 4.0 GiB VM-state, 435.2 MiB/s
2022-03-06 19:44:33 migration active, transferred 1.2 GiB of 4.0 GiB VM-state, 418.3 MiB/s
2022-03-06 19:44:34 migration active, transferred 1.6 GiB of 4.0 GiB VM-state, 417.6 MiB/s
2022-03-06 19:44:35 migration active, transferred 2.0 GiB of 4.0 GiB VM-state, 428.6 MiB/s
2022-03-06 19:44:36 migration active, transferred 2.4 GiB of 4.0 GiB VM-state, 442.4 MiB/s
2022-03-06 19:44:37 migration active, transferred 2.8 GiB of 4.0 GiB VM-state, 425.6 MiB/s
2022-03-06 19:44:38 migration active, transferred 3.2 GiB of 4.0 GiB VM-state, 420.7 MiB/s
2022-03-06 19:44:39 migration active, transferred 3.6 GiB of 4.0 GiB VM-state, 440.5 MiB/s
query migrate failed: VM 107 qmp command 'query-migrate' failed - client closed connection

2022-03-06 19:44:40 query migrate failed: VM 107 qmp command 'query-migrate' failed - client closed connection
query migrate failed: VM 107 not running

2022-03-06 19:44:41 query migrate failed: VM 107 not running
query migrate failed: VM 107 not running

2022-03-06 19:44:42 query migrate failed: VM 107 not running
query migrate failed: VM 107 not running

2022-03-06 19:44:43 query migrate failed: VM 107 not running
query migrate failed: VM 107 not running

2022-03-06 19:44:44 query migrate failed: VM 107 not running
query migrate failed: VM 107 not running

2022-03-06 19:44:45 query migrate failed: VM 107 not running
2022-03-06 19:44:45 ERROR: online migrate failure - too many query migrate failures - aborting
2022-03-06 19:44:45 aborting phase 2 - cleanup resources
2022-03-06 19:44:45 migrate_cancel
2022-03-06 19:44:45 migrate_cancel error: VM 107 not running
drive-virtio0: Cancelling block job
2022-03-06 19:44:45 ERROR: VM 107 not running
2022-03-06 19:44:54 ERROR: migration finished with problems (duration 00:09:17)
TASK ERROR: migration problems

 

[fiona]

Hi,
please post the output of qm config 107 and, for both source and target node, pveversion -v. Do you see anything interesting in /var/log/syslog (on the target, but checking the source too shouldn't hurt either) around the time the error happens?

 

[Helmo]

A similar thread, with more info, can be found in https://forum.proxmox.com/threads/live-migration-failed.111831/#post-482425