VMs migration failed. Any idea

E

ednt

Renowned Member
Mar 16, 2017
124
8
83
Today I needed to upgrade our last 8.4 PVE (latest version, all updates applied).
So I migrated all VMs to an already updated node (9.1.9)
All disks are on ceph storage.

2 window VMs resulted in a migration error and the VMs was not started on the new node.
I had to start them by hand, then they worked normal.

Log:
Code: 
2026-04-30 08:37:54 use dedicated network address for sending migration traffic (10.10.1.182)
2026-04-30 08:37:54 starting migration of VM 312 to node 'pm-cpu-182' (10.10.1.182)
2026-04-30 08:37:54 starting VM 312 on remote node 'pm-cpu-182'
2026-04-30 08:37:58 start remote tunnel
2026-04-30 08:38:00 ssh tunnel ver 1
2026-04-30 08:38:00 starting online/live migration on unix:/run/qemu-server/312.migrate
2026-04-30 08:38:00 set migration capabilities
2026-04-30 08:38:00 migration downtime limit: 100 ms
2026-04-30 08:38:00 migration cachesize: 8.0 GiB
2026-04-30 08:38:00 set migration parameters
2026-04-30 08:38:00 start migrate command to unix:/run/qemu-server/312.migrate
2026-04-30 08:38:01 migration active, transferred 385.3 MiB of 48.0 GiB VM-state, 420.0 MiB/s
:
:
:
2026-04-30 08:39:44 migration active, transferred 42.4 GiB of 48.0 GiB VM-state, 409.4 MiB/s
2026-04-30 08:39:44 average migration speed: 472.8 MiB/s - downtime 66 ms
2026-04-30 08:39:44 migration completed, transferred 42.4 GiB VM-state
2026-04-30 08:39:44 migration status: completed
2026-04-30 08:39:45 ERROR: tunnel replied 'ERR: resume failed - VM 312 qmp command 'query-status' failed - client closed connection' to command 'resume 312'
VM quit/powerdown failed - terminating now with SIGTERM
2026-04-30 08:39:59 ERROR: migration finished with problems (duration 00:02:06)
TASK ERROR: migration problems

Both VMs showed the same error.
On both windows servers are the qemu-guest-agent installed and running.
Any idea what happened and how it can be avoided?
 
Last edited:
please post the VM configs, full task logs of migration and start tasks as well as the journal from both nodes covering the migration.
 
Could this be similar to the issue discussed in the following thread?
This information might be helpful:

F

Thread 'Error migration conntrack in live mode'

It seemed that I had solved in the live migration of the VM but it repeats this rorry that I do not understand .... Sometimes it signals me that the conntrack is disabled sometimes not !
Code: 
2026-02-24 16:44:52 migration active, transferred 4.1 GiB of 8.0 GiB VM-state, 110.2 MiB/s
2026-02-24 16:44:53 migration active, transferred 4.2 GiB of 8.0 GiB VM-state, 137.1 MiB/s
2026-02-24 16:44:54 migration active, transferred 4.3 GiB of 8.0 GiB VM-state, 115.3 MiB/s
2026-02-24 16:44:55 migration active, transferred 4.4 GiB of 8.0 GiB VM-state, 116.7 MiB/s
2026-02-24 16:44:56 migration active...

As in that thread, CPU compatibility could be one possible cause. However, since this happened during a rolling upgrade, it may also be related to the mixed-version environment.

In either case, I would recommend checking the detailed logs with `journalctl`, not just the task log.
 
config:

Code: 
agent: 1
args: -cpu host,hypervisor=off
balloon: 0
boot: order=virtio1;ide0
cores: 4
cpu: x86-64-v2-AES
ide0: none,media=cdrom
machine: pc-i440fx-9.2+pve1
memory: 49152
name: ZIREC-SRV-APPL02
net0: virtio=16:6C:96:27:EE:65,bridge=vmbr1,tag=3005
net1: virtio=BC:24:11:7E:61:AD,bridge=vmbr1,firewall=1,tag=3004
numa: 0
onboot: 1
ostype: win10
parent: Backup_20260429
scsihw: virtio-scsi-single
smbios1: uuid=e6a27d0e-ebf1-4ad6-82fe-7e1fd9357279
sockets: 2
tags: autosnapshot
virtio0: ssd:vm-312-disk-1,cache=writeback,iothread=1,size=170G
virtio1: ssd:vm-312-disk-0,cache=writeback,iothread=1,size=450G
vmgenid: 31a82cd7-4435-417a-ae98-e5182c6f1e3e

[Backup_20260428]
agent: 1
args: -cpu host,hypervisor=off
balloon: 0
boot: order=virtio1;ide0
cores: 4
cpu: x86-64-v2-AES
ide0: none,media=cdrom
machine: pc-i440fx-9.2+pve1
memory: 49152
name: ZIREC-SRV-APPL02
net0: virtio=16:6C:96:27:EE:65,bridge=vmbr1,tag=3005
net1: virtio=BC:24:11:7E:61:AD,bridge=vmbr1,firewall=1,tag=3004
numa: 0
onboot: 1
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=e6a27d0e-ebf1-4ad6-82fe-7e1fd9357279
snaptime: 1777402932
sockets: 2
tags: autosnapshot
virtio0: ssd:vm-312-disk-1,cache=writeback,iothread=1,size=170G
virtio1: ssd:vm-312-disk-0,cache=writeback,iothread=1,size=450G
vmgenid: 31a82cd7-4435-417a-ae98-e5182c6f1e3e

[Backup_20260429]
agent: 1
args: -cpu host,hypervisor=off
balloon: 0
boot: order=virtio1;ide0
cores: 4
cpu: x86-64-v2-AES
ide0: none,media=cdrom
machine: pc-i440fx-9.2+pve1
memory: 49152
name: ZIREC-SRV-APPL02
net0: virtio=16:6C:96:27:EE:65,bridge=vmbr1,tag=3005
net1: virtio=BC:24:11:7E:61:AD,bridge=vmbr1,firewall=1,tag=3004
numa: 0
onboot: 1
ostype: win10
parent: Backup_20260428
scsihw: virtio-scsi-single
smbios1: uuid=e6a27d0e-ebf1-4ad6-82fe-7e1fd9357279
snaptime: 1777489332
sockets: 2
tags: autosnapshot
virtio0: ssd:vm-312-disk-1,cache=writeback,iothread=1,size=170G
virtio1: ssd:vm-312-disk-0,cache=writeback,iothread=1,size=450G
vmgenid: 31a82cd7-4435-417a-ae98-e5182c6f1e3e

Full log:
Code: 
2026-04-30 08:37:54 use dedicated network address for sending migration traffic (10.10.1.182)
2026-04-30 08:37:54 starting migration of VM 312 to node 'pm-cpu-182' (10.10.1.182)
2026-04-30 08:37:54 starting VM 312 on remote node 'pm-cpu-182'
2026-04-30 08:37:58 start remote tunnel
2026-04-30 08:38:00 ssh tunnel ver 1
2026-04-30 08:38:00 starting online/live migration on unix:/run/qemu-server/312.migrate
2026-04-30 08:38:00 set migration capabilities
2026-04-30 08:38:00 migration downtime limit: 100 ms
2026-04-30 08:38:00 migration cachesize: 8.0 GiB
2026-04-30 08:38:00 set migration parameters
2026-04-30 08:38:00 start migrate command to unix:/run/qemu-server/312.migrate
2026-04-30 08:38:01 migration active, transferred 385.3 MiB of 48.0 GiB VM-state, 420.0 MiB/s
2026-04-30 08:38:02 migration active, transferred 791.4 MiB of 48.0 GiB VM-state, 507.4 MiB/s
2026-04-30 08:38:03 migration active, transferred 1.2 GiB of 48.0 GiB VM-state, 577.0 MiB/s
2026-04-30 08:38:04 migration active, transferred 1.6 GiB of 48.0 GiB VM-state, 540.9 MiB/s
2026-04-30 08:38:05 migration active, transferred 2.0 GiB of 48.0 GiB VM-state, 424.8 MiB/s
2026-04-30 08:38:06 migration active, transferred 2.4 GiB of 48.0 GiB VM-state, 492.8 MiB/s
2026-04-30 08:38:07 migration active, transferred 2.8 GiB of 48.0 GiB VM-state, 1.6 GiB/s
2026-04-30 08:38:08 migration active, transferred 3.2 GiB of 48.0 GiB VM-state, 492.9 MiB/s
2026-04-30 08:38:09 migration active, transferred 3.6 GiB of 48.0 GiB VM-state, 476.0 MiB/s
2026-04-30 08:38:10 migration active, transferred 4.0 GiB of 48.0 GiB VM-state, 437.6 MiB/s
2026-04-30 08:38:11 migration active, transferred 4.4 GiB of 48.0 GiB VM-state, 451.5 MiB/s
2026-04-30 08:38:12 migration active, transferred 4.8 GiB of 48.0 GiB VM-state, 483.1 MiB/s
2026-04-30 08:38:13 migration active, transferred 5.2 GiB of 48.0 GiB VM-state, 461.2 MiB/s
2026-04-30 08:38:14 migration active, transferred 5.6 GiB of 48.0 GiB VM-state, 567.4 MiB/s
2026-04-30 08:38:15 migration active, transferred 6.0 GiB of 48.0 GiB VM-state, 464.0 MiB/s
2026-04-30 08:38:16 migration active, transferred 6.4 GiB of 48.0 GiB VM-state, 449.6 MiB/s
2026-04-30 08:38:17 migration active, transferred 6.8 GiB of 48.0 GiB VM-state, 492.9 MiB/s
2026-04-30 08:38:18 migration active, transferred 7.2 GiB of 48.0 GiB VM-state, 492.8 MiB/s
2026-04-30 08:38:19 migration active, transferred 7.6 GiB of 48.0 GiB VM-state, 492.8 MiB/s
2026-04-30 08:38:20 migration active, transferred 8.0 GiB of 48.0 GiB VM-state, 425.6 MiB/s
2026-04-30 08:38:21 migration active, transferred 8.5 GiB of 48.0 GiB VM-state, 442.4 MiB/s
2026-04-30 08:38:22 migration active, transferred 8.9 GiB of 48.0 GiB VM-state, 490.4 MiB/s
2026-04-30 08:38:23 migration active, transferred 9.3 GiB of 48.0 GiB VM-state, 449.6 MiB/s
2026-04-30 08:38:24 migration active, transferred 9.7 GiB of 48.0 GiB VM-state, 461.6 MiB/s
2026-04-30 08:38:25 migration active, transferred 10.1 GiB of 48.0 GiB VM-state, 468.8 MiB/s
2026-04-30 08:38:26 migration active, transferred 10.5 GiB of 48.0 GiB VM-state, 454.4 MiB/s
2026-04-30 08:38:27 migration active, transferred 10.9 GiB of 48.0 GiB VM-state, 446.7 MiB/s
2026-04-30 08:38:28 migration active, transferred 11.3 GiB of 48.0 GiB VM-state, 450.1 MiB/s
2026-04-30 08:38:29 migration active, transferred 11.7 GiB of 48.0 GiB VM-state, 445.3 MiB/s
2026-04-30 08:38:30 migration active, transferred 12.1 GiB of 48.0 GiB VM-state, 447.2 MiB/s
2026-04-30 08:38:31 migration active, transferred 12.5 GiB of 48.0 GiB VM-state, 446.7 MiB/s
2026-04-30 08:38:32 migration active, transferred 12.9 GiB of 48.0 GiB VM-state, 430.4 MiB/s
2026-04-30 08:38:33 migration active, transferred 13.3 GiB of 48.0 GiB VM-state, 454.0 MiB/s
2026-04-30 08:38:34 migration active, transferred 13.7 GiB of 48.0 GiB VM-state, 419.1 MiB/s
2026-04-30 08:38:35 migration active, transferred 14.1 GiB of 48.0 GiB VM-state, 441.8 MiB/s
2026-04-30 08:38:36 migration active, transferred 14.5 GiB of 48.0 GiB VM-state, 447.2 MiB/s
2026-04-30 08:38:37 migration active, transferred 14.9 GiB of 48.0 GiB VM-state, 541.3 MiB/s
2026-04-30 08:38:38 migration active, transferred 15.3 GiB of 48.0 GiB VM-state, 447.2 MiB/s
2026-04-30 08:38:39 migration active, transferred 15.7 GiB of 48.0 GiB VM-state, 456.4 MiB/s
2026-04-30 08:38:40 migration active, transferred 16.1 GiB of 48.0 GiB VM-state, 428.0 MiB/s
2026-04-30 08:38:41 migration active, transferred 16.5 GiB of 48.0 GiB VM-state, 459.2 MiB/s
2026-04-30 08:38:42 migration active, transferred 17.0 GiB of 48.0 GiB VM-state, 437.6 MiB/s
2026-04-30 08:38:43 migration active, transferred 17.4 GiB of 48.0 GiB VM-state, 500.1 MiB/s
2026-04-30 08:38:44 migration active, transferred 17.8 GiB of 48.0 GiB VM-state, 439.3 MiB/s
2026-04-30 08:38:45 migration active, transferred 18.2 GiB of 48.0 GiB VM-state, 449.6 MiB/s
2026-04-30 08:38:46 migration active, transferred 18.6 GiB of 48.0 GiB VM-state, 423.1 MiB/s
2026-04-30 08:38:47 migration active, transferred 19.0 GiB of 48.0 GiB VM-state, 488.1 MiB/s
2026-04-30 08:38:48 migration active, transferred 19.4 GiB of 48.0 GiB VM-state, 469.1 MiB/s
2026-04-30 08:38:49 migration active, transferred 19.8 GiB of 48.0 GiB VM-state, 446.2 MiB/s
2026-04-30 08:38:50 migration active, transferred 20.2 GiB of 48.0 GiB VM-state, 466.1 MiB/s
2026-04-30 08:38:51 migration active, transferred 20.6 GiB of 48.0 GiB VM-state, 447.2 MiB/s
2026-04-30 08:38:52 migration active, transferred 21.0 GiB of 48.0 GiB VM-state, 437.0 MiB/s
2026-04-30 08:38:53 migration active, transferred 21.4 GiB of 48.0 GiB VM-state, 454.0 MiB/s
2026-04-30 08:38:54 migration active, transferred 21.8 GiB of 48.0 GiB VM-state, 512.1 MiB/s
2026-04-30 08:38:55 migration active, transferred 22.2 GiB of 48.0 GiB VM-state, 432.8 MiB/s
2026-04-30 08:38:56 migration active, transferred 22.6 GiB of 48.0 GiB VM-state, 435.2 MiB/s
2026-04-30 08:38:57 migration active, transferred 23.0 GiB of 48.0 GiB VM-state, 446.7 MiB/s
2026-04-30 08:38:58 migration active, transferred 23.5 GiB of 48.0 GiB VM-state, 419.1 MiB/s
2026-04-30 08:38:59 migration active, transferred 23.9 GiB of 48.0 GiB VM-state, 422.4 MiB/s
2026-04-30 08:39:00 migration active, transferred 24.3 GiB of 48.0 GiB VM-state, 400.1 MiB/s
2026-04-30 08:39:01 migration active, transferred 24.7 GiB of 48.0 GiB VM-state, 473.6 MiB/s
2026-04-30 08:39:02 migration active, transferred 25.1 GiB of 48.0 GiB VM-state, 473.6 MiB/s
2026-04-30 08:39:03 migration active, transferred 25.5 GiB of 48.0 GiB VM-state, 478.2 MiB/s
2026-04-30 08:39:04 migration active, transferred 26.0 GiB of 48.0 GiB VM-state, 452.0 MiB/s
2026-04-30 08:39:05 migration active, transferred 26.4 GiB of 48.0 GiB VM-state, 428.0 MiB/s
2026-04-30 08:39:06 migration active, transferred 26.8 GiB of 48.0 GiB VM-state, 446.7 MiB/s
2026-04-30 08:39:07 migration active, transferred 27.2 GiB of 48.0 GiB VM-state, 466.7 MiB/s
2026-04-30 08:39:08 migration active, transferred 27.6 GiB of 48.0 GiB VM-state, 475.8 MiB/s
2026-04-30 08:39:09 migration active, transferred 28.1 GiB of 48.0 GiB VM-state, 464.0 MiB/s
2026-04-30 08:39:10 migration active, transferred 28.5 GiB of 48.0 GiB VM-state, 473.6 MiB/s
2026-04-30 08:39:11 migration active, transferred 28.9 GiB of 48.0 GiB VM-state, 500.1 MiB/s
2026-04-30 08:39:12 migration active, transferred 29.3 GiB of 48.0 GiB VM-state, 432.8 MiB/s
2026-04-30 08:39:13 migration active, transferred 29.7 GiB of 48.0 GiB VM-state, 435.2 MiB/s
2026-04-30 08:39:14 migration active, transferred 30.2 GiB of 48.0 GiB VM-state, 466.4 MiB/s
2026-04-30 08:39:15 migration active, transferred 30.6 GiB of 48.0 GiB VM-state, 473.9 MiB/s
2026-04-30 08:39:16 migration active, transferred 31.0 GiB of 48.0 GiB VM-state, 514.6 MiB/s
2026-04-30 08:39:17 migration active, transferred 31.4 GiB of 48.0 GiB VM-state, 431.4 MiB/s
2026-04-30 08:39:18 migration active, transferred 31.8 GiB of 48.0 GiB VM-state, 500.1 MiB/s
2026-04-30 08:39:19 migration active, transferred 32.3 GiB of 48.0 GiB VM-state, 514.5 MiB/s
2026-04-30 08:39:20 migration active, transferred 32.7 GiB of 48.0 GiB VM-state, 516.9 MiB/s
2026-04-30 08:39:21 migration active, transferred 33.1 GiB of 48.0 GiB VM-state, 463.7 MiB/s
2026-04-30 08:39:22 migration active, transferred 33.5 GiB of 48.0 GiB VM-state, 446.7 MiB/s
2026-04-30 08:39:23 migration active, transferred 33.9 GiB of 48.0 GiB VM-state, 689.4 MiB/s
2026-04-30 08:39:24 migration active, transferred 34.4 GiB of 48.0 GiB VM-state, 447.2 MiB/s
2026-04-30 08:39:25 migration active, transferred 34.8 GiB of 48.0 GiB VM-state, 519.3 MiB/s
2026-04-30 08:39:26 migration active, transferred 35.2 GiB of 48.0 GiB VM-state, 509.8 MiB/s
2026-04-30 08:39:27 migration active, transferred 35.6 GiB of 48.0 GiB VM-state, 507.4 MiB/s
2026-04-30 08:39:28 migration active, transferred 36.0 GiB of 48.0 GiB VM-state, 447.2 MiB/s
2026-04-30 08:39:29 migration active, transferred 36.5 GiB of 48.0 GiB VM-state, 490.5 MiB/s
2026-04-30 08:39:30 migration active, transferred 36.9 GiB of 48.0 GiB VM-state, 426.3 MiB/s
2026-04-30 08:39:31 migration active, transferred 37.3 GiB of 48.0 GiB VM-state, 478.2 MiB/s
2026-04-30 08:39:32 migration active, transferred 37.7 GiB of 48.0 GiB VM-state, 444.8 MiB/s
2026-04-30 08:39:33 migration active, transferred 38.1 GiB of 48.0 GiB VM-state, 507.3 MiB/s
2026-04-30 08:39:34 migration active, transferred 38.5 GiB of 48.0 GiB VM-state, 479.4 MiB/s
2026-04-30 08:39:35 migration active, transferred 38.9 GiB of 48.0 GiB VM-state, 497.7 MiB/s
2026-04-30 08:39:36 migration active, transferred 39.3 GiB of 48.0 GiB VM-state, 509.9 MiB/s
2026-04-30 08:39:37 migration active, transferred 39.7 GiB of 48.0 GiB VM-state, 509.8 MiB/s
2026-04-30 08:39:38 migration active, transferred 40.1 GiB of 48.0 GiB VM-state, 463.3 MiB/s
2026-04-30 08:39:39 migration active, transferred 40.5 GiB of 48.0 GiB VM-state, 473.4 MiB/s
2026-04-30 08:39:40 migration active, transferred 40.9 GiB of 48.0 GiB VM-state, 483.2 MiB/s
2026-04-30 08:39:41 migration active, transferred 41.3 GiB of 48.0 GiB VM-state, 483.1 MiB/s
2026-04-30 08:39:43 migration active, transferred 42.0 GiB of 48.0 GiB VM-state, 563.2 MiB/s
2026-04-30 08:39:44 migration active, transferred 42.4 GiB of 48.0 GiB VM-state, 409.4 MiB/s
2026-04-30 08:39:44 average migration speed: 472.8 MiB/s - downtime 66 ms
2026-04-30 08:39:44 migration completed, transferred 42.4 GiB VM-state
2026-04-30 08:39:44 migration status: completed
2026-04-30 08:39:45 ERROR: tunnel replied 'ERR: resume failed - VM 312 qmp command 'query-status' failed - client closed connection' to command 'resume 312'
VM quit/powerdown failed - terminating now with SIGTERM
2026-04-30 08:39:59 ERROR: migration finished with problems (duration 00:02:06)
TASK ERROR: migration problems

Start at begin of migration:
Code: 
netdev net0: not adding 'host_mtu' parameter for migration compat
netdev net1: not adding 'host_mtu' parameter for migration compat
migration listens on unix:/run/qemu-server/312.migrate
TASK OK

My manual start after 'failed'migration:
Code: 
TASK OK

There are 2 snapshots available.
 
You must log in or register to reply here.
Top Bottom
Back