Failed to complete storage migration: block job (mirror) error

[cglmicro]

Hello,

I'm trying to migrate a VM from one node to another in the same cluster, and it fail at the end with this error:

2025-04-09 10:23:55 use dedicated network address for sending migration traffic (192.168.150.26)
2025-04-09 10:23:55 starting migration of VM 11011 to node 'proxmox26s' (192.168.150.26)
2025-04-09 10:23:55 found local disk 'local:11011/vm-11011-disk-0.qcow2' (attached)
2025-04-09 10:23:55 found local disk 'local:11011/vm-11011-disk-1.qcow2' (attached)
2025-04-09 10:23:55 starting VM 11011 on remote node 'proxmox26s'
2025-04-09 10:23:57 volume 'local:11011/vm-11011-disk-0.qcow2' is 'local:11011/vm-11011-disk-0.qcow2' on the target
2025-04-09 10:23:57 volume 'local:11011/vm-11011-disk-1.qcow2' is 'local:11011/vm-11011-disk-1.qcow2' on the target
2025-04-09 10:23:57 start remote tunnel
2025-04-09 10:23:58 ssh tunnel ver 1
2025-04-09 10:23:58 starting storage migration
2025-04-09 10:23:58 scsi1: start migration to nbd:unix:/run/qemu-server/11011_nbd.migrate:exportname=drive-scsi1
drive mirror is starting for drive-scsi1
drive-scsi1: transferred 196.9 MiB of 50.0 GiB (0.38%) in 1s
... skipped 170 lines...
drive-scsi1: transferred 50.2 GiB of 50.2 GiB (100.00%) in 2m 55s, ready
all 'mirror' jobs are ready
2025-04-09 10:26:53 scsi0: start migration to nbd:unix:/run/qemu-server/11011_nbd.migrate:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 124.9 MiB of 325.0 GiB (0.04%) in 12s
... skipped 1610 lines...
drive-scsi0: transferred 325.2 GiB of 325.2 GiB (100.00%) in 27m 13s, ready
all 'mirror' jobs are ready
2025-04-09 10:54:06 switching mirror jobs to actively synced mode
drive-scsi0: switching to actively synced mode
drive-scsi1: switching to actively synced mode
drive-scsi0: successfully switched to actively synced mode
drive-scsi1: successfully switched to actively synced mode
2025-04-09 10:54:07 starting online/live migration on unix:/run/qemu-server/11011.migrate
2025-04-09 10:54:07 set migration capabilities
2025-04-09 10:54:07 migration downtime limit: 100 ms
2025-04-09 10:54:07 migration cachesize: 2.0 GiB
2025-04-09 10:54:07 set migration parameters
2025-04-09 10:54:07 start migrate command to unix:/run/qemu-server/11011.migrate
2025-04-09 10:54:08 migration active, transferred 134.9 MiB of 18.0 GiB VM-state, 119.3 MiB/s
... skipped 165 lines...
2025-04-09 10:56:56 migration active, transferred 18.6 GiB of 18.0 GiB VM-state, 122.7 MiB/s
2025-04-09 10:56:57 migration active, transferred 18.7 GiB of 18.0 GiB VM-state, 114.4 MiB/s
2025-04-09 10:56:58 migration active, transferred 18.8 GiB of 18.0 GiB VM-state, 110.9 MiB/s
2025-04-09 10:56:59 migration active, transferred 18.9 GiB of 18.0 GiB VM-state, 111.7 MiB/s
2025-04-09 10:57:00 migration active, transferred 19.0 GiB of 18.0 GiB VM-state, 123.9 MiB/s
2025-04-09 10:57:01 migration active, transferred 19.1 GiB of 18.0 GiB VM-state, 115.6 MiB/s
2025-04-09 10:57:02 migration active, transferred 19.2 GiB of 18.0 GiB VM-state, 116.6 MiB/s
2025-04-09 10:57:03 migration active, transferred 19.3 GiB of 18.0 GiB VM-state, 15.7 MiB/s, VM dirties lots of memory: 62.9 MiB/s
2025-04-09 10:57:04 migration active, transferred 19.4 GiB of 18.0 GiB VM-state, 112.2 MiB/s
2025-04-09 10:57:05 migration active, transferred 19.6 GiB of 18.0 GiB VM-state, 114.4 MiB/s
2025-04-09 10:57:06 migration active, transferred 19.7 GiB of 18.0 GiB VM-state, 115.5 MiB/s
2025-04-09 10:57:07 migration active, transferred 19.8 GiB of 18.0 GiB VM-state, 111.2 MiB/s
2025-04-09 10:57:08 migration active, transferred 19.9 GiB of 18.0 GiB VM-state, 109.9 MiB/s
2025-04-09 10:57:09 migration active, transferred 20.0 GiB of 18.0 GiB VM-state, 109.3 MiB/s
2025-04-09 10:57:10 migration active, transferred 20.1 GiB of 18.0 GiB VM-state, 140.1 MiB/s
2025-04-09 10:57:10 xbzrle: send updates to 14120 pages in 20.4 MiB encoded memory, cache-miss 99.10%, overflow 1361
2025-04-09 10:57:11 migration active, transferred 20.2 GiB of 18.0 GiB VM-state, 142.4 MiB/s
2025-04-09 10:57:11 xbzrle: send updates to 25842 pages in 50.9 MiB encoded memory, cache-miss 99.10%, overflow 2803
2025-04-09 10:57:12 migration active, transferred 20.3 GiB of 18.0 GiB VM-state, 135.7 MiB/s
2025-04-09 10:57:12 xbzrle: send updates to 37807 pages in 82.2 MiB encoded memory, cache-miss 99.10%, overflow 4290
2025-04-09 10:57:13 migration active, transferred 20.4 GiB of 18.0 GiB VM-state, 134.0 MiB/s
2025-04-09 10:57:13 xbzrle: send updates to 48124 pages in 108.6 MiB encoded memory, cache-miss 75.58%, overflow 5518
2025-04-09 10:57:13 auto-increased downtime to continue migration: 200 ms
2025-04-09 10:57:14 migration active, transferred 20.5 GiB of 18.0 GiB VM-state, 128.4 MiB/s
2025-04-09 10:57:14 xbzrle: send updates to 57719 pages in 135.9 MiB encoded memory, cache-miss 75.58%, overflow 6849
2025-04-09 10:57:15 migration active, transferred 20.7 GiB of 18.0 GiB VM-state, 133.0 MiB/s
2025-04-09 10:57:15 xbzrle: send updates to 69871 pages in 169.5 MiB encoded memory, cache-miss 75.58%, overflow 8582
2025-04-09 10:57:16 migration active, transferred 20.8 GiB of 18.0 GiB VM-state, 139.9 MiB/s
2025-04-09 10:57:16 xbzrle: send updates to 79971 pages in 194.5 MiB encoded memory, cache-miss 75.58%, overflow 9929
2025-04-09 10:57:17 migration active, transferred 20.9 GiB of 18.0 GiB VM-state, 160.3 MiB/s
2025-04-09 10:57:17 xbzrle: send updates to 97533 pages in 236.9 MiB encoded memory, cache-miss 75.58%, overflow 12522
2025-04-09 10:57:18 migration active, transferred 21.0 GiB of 18.0 GiB VM-state, 164.2 MiB/s
2025-04-09 10:57:18 xbzrle: send updates to 115012 pages in 272.2 MiB encoded memory, cache-miss 64.16%, overflow 15182
2025-04-09 10:57:19 migration active, transferred 21.1 GiB of 18.0 GiB VM-state, 805.5 MiB/s
2025-04-09 10:57:19 xbzrle: send updates to 168193 pages in 293.6 MiB encoded memory, cache-miss 64.16%, overflow 16554
2025-04-09 10:57:20 average migration speed: 95.6 MiB/s - downtime 58 ms
2025-04-09 10:57:20 migration status: completed
all 'mirror' jobs are ready
drive-scsi0: Completing block job...
drive-scsi0: Completed successfully.
drive-scsi1: Completing block job...
drive-scsi1: Completed successfully.
drive-scsi0: Cancelling block job
drive-scsi1: Cancelling block job
drive-scsi0: Done.
WARN: drive-scsi1: Input/output error (io-status: ok)
drive-scsi1: Done.
2025-04-09 10:57:22 ERROR: online migrate failure - Failed to complete storage migration: block job (mirror) error: drive-scsi0: Input/output error (io-status: ok)
2025-04-09 10:57:22 aborting phase 2 - cleanup resources
2025-04-09 10:57:22 migrate_cancel
2025-04-09 10:57:28 ERROR: migration finished with problems (duration 00:33:33)
TASK ERROR: migration problems

There is no snapshot, no TPM, and here are the spec of the VM:


Should I try to plan a downtime of 30 minutes, shut down the VM and try the migration again with a powered off VM?

Your help would be appreciated
Thank you.

 

[fiona]

Hi,
please share the output of pveversion -v as well as the system log/journal from around the time the issue happened, from both source node and target node. Please also share the output of qm config 11011.

 

[cglmicro]

Hello Fiona,

Thank you for your reply. I tried it once again yesterday after updating from PVE 8.3.x to 8.4.0, same problem.

You will find both logs attached to this ticket.

Here is the config of the VM:

root@proxmox24s:~# qm config 11011
agent: 1,fstrim_cloned_disks=1
bootdisk: scsi0
cores: 9
cpu: host
ide2: none,media=cdrom
memory: 18432
name: ssd2.legardeur.net
net0: virtio=02:00:00:6d:6a:f9,bridge=vmbr0
numa: 1
onboot: 1
ostype: l26
protection: 1
scsi0: local:11011/vm-11011-disk-0.qcow2,discard=on,format=qcow2,size=325G
scsi1: local:11011/vm-11011-disk-1.qcow2,discard=on,size=50G
scsihw: virtio-scsi-pci
smbios1: uuid=91f02b56-3f07-45c3-95b2-b35470e9a540
sockets: 1
vmgenid: 03d21b16-1578-4f87-93c2-2d6915a6faf8

From the source (PROXMOX24S):

root@proxmox24s:~# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-8-pve)
pve-manager: 8.4.0 (running version: 8.4.0/ec58e45e1bcdf2ac)
proxmox-kernel-helper: 8.1.1
pve-kernel-6.2: 8.0.5
pve-kernel-5.15: 7.4-4
proxmox-kernel-6.8: 6.8.12-9
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
proxmox-kernel-6.8.12-3-pve-signed: 6.8.12-3
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
proxmox-kernel-6.8.8-3-pve-signed: 6.8.8-3
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
proxmox-kernel-6.2.16-12-pve: 6.2.16-12
proxmox-kernel-6.2.16-10-pve: 6.2.16-10
proxmox-kernel-6.2.16-8-pve: 6.2.16-8
proxmox-kernel-6.2.16-6-pve: 6.2.16-7
pve-kernel-6.2.16-4-pve: 6.2.16-5
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.107-1-pve: 5.15.107-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.3.7-1
proxmox-backup-file-restore: 3.3.7-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.3
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2

From the destination (PROXMOX26S):

root@proxmox26s:~# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-8-pve)
pve-manager: 8.4.0 (running version: 8.4.0/ec58e45e1bcdf2ac)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8: 6.8.12-9
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.3.7-1
proxmox-backup-file-restore: 3.3.7-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.3
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2

Thank you

 

[fiona]

On the target, there is the following error:

QEMU[1710235]: kvm: Putting registers after init: Failed to set special registers: Invalid argument

You are using CPU type host for the VM. In that case live migration can only be guaranteed to work if you have the exact same physical CPU model on both source and target side. You can check with lscpu, sometimes it's even necessary to have the same microcode version too.
See: https://pve.proxmox.com/pve-docs/chapter-qm.html#_cpu_type

There's also lots of messages from Corosync. Do you use the same network for migration and Corosync? That is not recommended as Corosync requires low latency: