Hi everybody,
I built a small cluster with 3 hosts. Each of them has a ext4 partition mounted to /raidsystem used as storage for the VMs disks.
If I shutdown a VM and migrate it from one host to another, I get a failure:
If I do the same with VM running (life migration), it works:
Further, life migration seems to be much faster. Might I make any weird mistake? I do not find anything helpful in the server log files. If I can provide anything that might help, please let me know.
Regards,
Fant
I built a small cluster with 3 hosts. Each of them has a ext4 partition mounted to /raidsystem used as storage for the VMs disks.
If I shutdown a VM and migrate it from one host to another, I get a failure:
Code:
2026-01-23 13:21:48 starting migration of VM 99998 to node 'proxmox3' (192.168.1.58)
2026-01-23 13:21:48 found local disk 'raidsystem:99998/vm-99998-disk-0.qcow2' (attached)
2026-01-23 13:21:48 copying local disk images
2026-01-23 13:21:50 Formatting '/raidsystem/images/99998/vm-99998-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=34359738368 lazy_refcounts=off refcount_bits=16
2026-01-23 13:21:53 98144256 bytes (98 MB, 94 MiB) copied, 3 s, 32.6 MB/s
2026-01-23 13:21:56 269488128 bytes (269 MB, 257 MiB) copied, 6 s, 44.9 MB/s
2026-01-23 13:21:59 439521280 bytes (440 MB, 419 MiB) copied, 9 s, 48.8 MB/s
2026-01-23 13:22:02 611061760 bytes (611 MB, 583 MiB) copied, 12 s, 50.9 MB/s
2026-01-23 13:22:05 762941440 bytes (763 MB, 728 MiB) copied, 15 s, 50.1 MB/s
2026-01-23 13:22:08 919932928 bytes (920 MB, 877 MiB) copied, 18 s, 51.1 MB/s
2026-01-23 13:22:11 1090424832 bytes (1.1 GB, 1.0 GiB) copied, 21 s, 51.9 MB/s
2026-01-23 13:22:14 1257443328 bytes (1.3 GB, 1.2 GiB) copied, 24 s, 52.4 MB/s
2026-01-23 13:22:17 1404506112 bytes (1.4 GB, 1.3 GiB) copied, 27 s, 52.0 MB/s
2026-01-23 13:22:20 1564250112 bytes (1.6 GB, 1.5 GiB) copied, 30 s, 52.1 MB/s
2026-01-23 13:22:23 1713672192 bytes (1.7 GB, 1.6 GiB) copied, 33 s, 51.9 MB/s
2026-01-23 13:22:26 1806274560 bytes (1.8 GB, 1.7 GiB) copied, 36 s, 50.2 MB/s
2026-01-23 13:22:29 1928663040 bytes (1.9 GB, 1.8 GiB) copied, 39 s, 49.4 MB/s
2026-01-23 13:22:32 2087063552 bytes (2.1 GB, 1.9 GiB) copied, 42 s, 49.7 MB/s
2026-01-23 13:22:35 2262110208 bytes (2.3 GB, 2.1 GiB) copied, 45 s, 50.3 MB/s
2026-01-23 13:22:38 2437582848 bytes (2.4 GB, 2.3 GiB) copied, 48 s, 50.8 MB/s
2026-01-23 13:22:41 2612269056 bytes (2.6 GB, 2.4 GiB) copied, 51 s, 51.2 MB/s
2026-01-23 13:22:44 2786103296 bytes (2.8 GB, 2.6 GiB) copied, 54 s, 51.6 MB/s
2026-01-23 13:22:47 2961018880 bytes (3.0 GB, 2.8 GiB) copied, 57 s, 51.9 MB/s
2026-01-23 13:22:50 3136786432 bytes (3.1 GB, 2.9 GiB) copied, 60 s, 52.3 MB/s
2026-01-23 13:23:00 3710095360 bytes (3.7 GB, 3.5 GiB) copied, 70 s, 53.0 MB/s
2026-01-23 13:23:10 4270919680 bytes (4.3 GB, 4.0 GiB) copied, 80 s, 53.4 MB/s
2026-01-23 13:23:20 4833382400 bytes (4.8 GB, 4.5 GiB) copied, 90 s, 53.7 MB/s
2026-01-23 13:23:30 5414948864 bytes (5.4 GB, 5.0 GiB) copied, 100 s, 54.1 MB/s
2026-01-23 13:23:40 5976199168 bytes (6.0 GB, 5.6 GiB) copied, 110 s, 54.3 MB/s
2026-01-23 13:23:50 6513332224 bytes (6.5 GB, 6.1 GiB) copied, 120 s, 54.3 MB/s
2026-01-23 13:24:00 7048171520 bytes (7.0 GB, 6.6 GiB) copied, 130 s, 54.2 MB/s
2026-01-23 13:24:10 7614787584 bytes (7.6 GB, 7.1 GiB) copied, 140 s, 54.4 MB/s
2026-01-23 13:24:20 8195870720 bytes (8.2 GB, 7.6 GiB) copied, 150 s, 54.6 MB/s
2026-01-23 13:24:30 8773865472 bytes (8.8 GB, 8.2 GiB) copied, 160 s, 54.8 MB/s
2026-01-23 13:24:40 9326333952 bytes (9.3 GB, 8.7 GiB) copied, 170 s, 54.9 MB/s
2026-01-23 13:24:50 9853865984 bytes (9.9 GB, 9.2 GiB) copied, 180 s, 54.7 MB/s
2026-01-23 13:25:00 10394013696 bytes (10 GB, 9.7 GiB) copied, 190 s, 54.7 MB/s
2026-01-23 13:25:10 10951659520 bytes (11 GB, 10 GiB) copied, 200 s, 54.8 MB/s
2026-01-23 13:25:20 11517169664 bytes (12 GB, 11 GiB) copied, 210 s, 54.8 MB/s
2026-01-23 13:25:30 12092542976 bytes (12 GB, 11 GiB) copied, 220 s, 55.0 MB/s
2026-01-23 13:25:40 12670111744 bytes (13 GB, 12 GiB) copied, 230 s, 55.1 MB/s
2026-01-23 13:41:38 client_loop: send disconnect: Broken pipe
2026-01-23 13:41:38 command 'dd 'if=/raidsystem/images/99998/vm-99998-disk-0.qcow2' 'bs=4k' 'status=progress'' failed: got signal 13
2026-01-23 13:41:38 ERROR: storage migration for 'raidsystem:99998/vm-99998-disk-0.qcow2' to storage 'raidsystem' failed - command 'set -o pipefail && pvesm export raidsystem:99998/vm-99998-disk-0.qcow2 qcow2+size - -with-snapshots 1 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=proxmox3' -o 'UserKnownHostsFile=/etc/pve/nodes/proxmox3/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' root@192.168.1.58 -- pvesm import raidsystem:99998/vm-99998-disk-0.qcow2 qcow2+size - -with-snapshots 1 -allow-rename 1' failed: exit code 255
2026-01-23 13:41:38 aborting phase 1 - cleanup resources
2026-01-23 13:41:38 ERROR: migration aborted (duration 00:19:50): storage migration for 'raidsystem:99998/vm-99998-disk-0.qcow2' to storage 'raidsystem' failed - command 'set -o pipefail && pvesm export raidsystem:99998/vm-99998-disk-0.qcow2 qcow2+size - -with-snapshots 1 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=proxmox3' -o 'UserKnownHostsFile=/etc/pve/nodes/proxmox3/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' root@192.168.1.58 -- pvesm import raidsystem:99998/vm-99998-disk-0.qcow2 qcow2+size - -with-snapshots 1 -allow-rename 1' failed: exit code 255
TASK ERROR: migration abortedIf I do the same with VM running (life migration), it works:
Code:
2026-01-23 14:16:54 starting migration of VM 99998 to node 'proxmox' (192.168.1.57)
2026-01-23 14:16:55 found local disk 'raidsystem:99998/vm-99998-disk-0.qcow2' (attached)
2026-01-23 14:16:55 starting VM 99998 on remote node 'proxmox'
2026-01-23 14:17:01 volume 'raidsystem:99998/vm-99998-disk-0.qcow2' is 'raidsystem:99998/vm-99998-disk-0.qcow2' on the target
2026-01-23 14:17:02 start remote tunnel
2026-01-23 14:17:04 ssh tunnel ver 1
2026-01-23 14:17:04 starting storage migration
2026-01-23 14:17:04 scsi0: start migration to nbd:unix:/run/qemu-server/99998_nbd.migrate:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
mirror-scsi0: transferred 134.0 MiB of 32.0 GiB (0.41%) in 1s
mirror-scsi0: transferred 189.0 MiB of 32.0 GiB (0.58%) in 2s
mirror-scsi0: transferred 243.0 MiB of 32.0 GiB (0.74%) in 3s
mirror-scsi0: transferred 297.0 MiB of 32.0 GiB (0.91%) in 4s
mirror-scsi0: transferred 352.0 MiB of 32.0 GiB (1.07%) in 5s
mirror-scsi0: transferred 411.5 MiB of 32.0 GiB (1.26%) in 6s
mirror-scsi0: transferred 465.0 MiB of 32.0 GiB (1.42%) in 7s
mirror-scsi0: transferred 519.0 MiB of 32.0 GiB (1.58%) in 8s
mirror-scsi0: transferred 573.2 MiB of 32.0 GiB (1.75%) in 9s
mirror-scsi0: transferred 627.0 MiB of 32.0 GiB (1.91%) in 10s
mirror-scsi0: transferred 685.0 MiB of 32.0 GiB (2.09%) in 11s
mirror-scsi0: transferred 739.0 MiB of 32.0 GiB (2.26%) in 12s
mirror-scsi0: transferred 794.0 MiB of 32.0 GiB (2.42%) in 13s
mirror-scsi0: transferred 848.0 MiB of 32.0 GiB (2.59%) in 14s
mirror-scsi0: transferred 907.0 MiB of 32.0 GiB (2.77%) in 15s
mirror-scsi0: transferred 961.0 MiB of 32.0 GiB (2.93%) in 16s
mirror-scsi0: transferred 1014.0 MiB of 32.0 GiB (3.09%) in 17s
mirror-scsi0: transferred 1.0 GiB of 32.0 GiB (3.26%) in 18s
mirror-scsi0: transferred 1.1 GiB of 32.0 GiB (3.42%) in 19s
mirror-scsi0: transferred 1.2 GiB of 32.0 GiB (3.61%) in 20s
mirror-scsi0: transferred 1.2 GiB of 32.0 GiB (3.79%) in 21s
mirror-scsi0: transferred 1.3 GiB of 32.0 GiB (3.97%) in 22s
mirror-scsi0: transferred 1.3 GiB of 32.0 GiB (4.14%) in 23s
mirror-scsi0: transferred 1.4 GiB of 32.0 GiB (4.31%) in 24s
mirror-scsi0: transferred 1.4 GiB of 32.0 GiB (4.50%) in 25s
mirror-scsi0: transferred 1.5 GiB of 32.0 GiB (4.67%) in 26s
mirror-scsi0: transferred 1.5 GiB of 32.0 GiB (4.83%) in 27s
mirror-scsi0: transferred 1.6 GiB of 32.0 GiB (5.00%) in 28s
mirror-scsi0: transferred 1.7 GiB of 32.0 GiB (5.17%) in 29s
mirror-scsi0: transferred 1.8 GiB of 32.0 GiB (5.51%) in 30s
mirror-scsi0: transferred 1.8 GiB of 32.0 GiB (5.67%) in 31s
mirror-scsi0: transferred 8.1 GiB of 32.0 GiB (25.22%) in 32s
mirror-scsi0: transferred 8.2 GiB of 32.0 GiB (25.56%) in 33s
mirror-scsi0: transferred 8.2 GiB of 32.0 GiB (25.73%) in 34s
mirror-scsi0: transferred 8.3 GiB of 32.0 GiB (25.91%) in 35s
mirror-scsi0: transferred 8.3 GiB of 32.0 GiB (26.08%) in 36s
mirror-scsi0: transferred 8.4 GiB of 32.0 GiB (26.25%) in 37s
mirror-scsi0: transferred 8.5 GiB of 32.0 GiB (26.43%) in 38s
mirror-scsi0: transferred 8.5 GiB of 32.0 GiB (26.60%) in 39s
mirror-scsi0: transferred 8.6 GiB of 32.0 GiB (26.77%) in 40s
mirror-scsi0: transferred 8.6 GiB of 32.0 GiB (26.94%) in 41s
mirror-scsi0: transferred 8.7 GiB of 32.0 GiB (27.12%) in 42s
mirror-scsi0: transferred 8.7 GiB of 32.0 GiB (27.29%) in 43s
mirror-scsi0: transferred 9.9 GiB of 32.0 GiB (30.97%) in 44s
mirror-scsi0: transferred 10.1 GiB of 32.0 GiB (31.68%) in 45s
mirror-scsi0: transferred 10.2 GiB of 32.0 GiB (31.86%) in 46s
mirror-scsi0: transferred 10.2 GiB of 32.0 GiB (32.03%) in 47s
mirror-scsi0: transferred 10.3 GiB of 32.0 GiB (32.20%) in 48s
mirror-scsi0: transferred 10.4 GiB of 32.0 GiB (32.38%) in 49s
mirror-scsi0: transferred 10.4 GiB of 32.0 GiB (32.64%) in 50s
mirror-scsi0: transferred 12.2 GiB of 32.0 GiB (37.98%) in 51s
mirror-scsi0: transferred 12.2 GiB of 32.0 GiB (38.16%) in 52s
mirror-scsi0: transferred 12.3 GiB of 32.0 GiB (38.34%) in 53s
mirror-scsi0: transferred 12.3 GiB of 32.0 GiB (38.52%) in 54s
mirror-scsi0: transferred 12.4 GiB of 32.0 GiB (38.71%) in 55s
mirror-scsi0: transferred 12.5 GiB of 32.0 GiB (38.95%) in 56s
mirror-scsi0: transferred 12.5 GiB of 32.0 GiB (39.17%) in 57s
mirror-scsi0: transferred 12.6 GiB of 32.0 GiB (39.38%) in 58s
mirror-scsi0: transferred 12.7 GiB of 32.0 GiB (39.56%) in 59s
mirror-scsi0: transferred 12.7 GiB of 32.0 GiB (39.74%) in 1m
mirror-scsi0: transferred 12.8 GiB of 32.0 GiB (39.94%) in 1m 1s
mirror-scsi0: transferred 12.8 GiB of 32.0 GiB (40.13%) in 1m 2s
mirror-scsi0: transferred 12.9 GiB of 32.0 GiB (40.31%) in 1m 3s
mirror-scsi0: transferred 13.0 GiB of 32.0 GiB (40.49%) in 1m 4s
mirror-scsi0: transferred 13.0 GiB of 32.0 GiB (40.73%) in 1m 5s
mirror-scsi0: transferred 14.2 GiB of 32.0 GiB (44.28%) in 1m 6s
mirror-scsi0: transferred 14.2 GiB of 32.0 GiB (44.45%) in 1m 7s
mirror-scsi0: transferred 17.4 GiB of 32.0 GiB (54.44%) in 1m 8s
mirror-scsi0: transferred 17.5 GiB of 32.0 GiB (54.62%) in 1m 9s
mirror-scsi0: transferred 17.5 GiB of 32.0 GiB (54.82%) in 1m 10s
mirror-scsi0: transferred 17.6 GiB of 32.0 GiB (55.00%) in 1m 11s
mirror-scsi0: transferred 17.7 GiB of 32.0 GiB (55.19%) in 1m 12s
mirror-scsi0: transferred 17.7 GiB of 32.0 GiB (55.36%) in 1m 13s
mirror-scsi0: transferred 17.8 GiB of 32.0 GiB (55.55%) in 1m 14s
mirror-scsi0: transferred 17.8 GiB of 32.0 GiB (55.73%) in 1m 15s
mirror-scsi0: transferred 17.9 GiB of 32.0 GiB (55.96%) in 1m 16s
mirror-scsi0: transferred 18.0 GiB of 32.0 GiB (56.13%) in 1m 17s
mirror-scsi0: transferred 18.1 GiB of 32.0 GiB (56.67%) in 1m 18s
mirror-scsi0: transferred 18.2 GiB of 32.0 GiB (56.86%) in 1m 19s
mirror-scsi0: transferred 18.5 GiB of 32.0 GiB (57.86%) in 1m 20s
mirror-scsi0: transferred 18.6 GiB of 32.0 GiB (58.15%) in 1m 21s
mirror-scsi0: transferred 21.2 GiB of 32.0 GiB (66.25%) in 1m 22s
mirror-scsi0: transferred 32.0 GiB of 32.0 GiB (100.00%) in 1m 23s, ready
all 'mirror' jobs are ready
2026-01-23 14:18:27 switching mirror jobs to actively synced mode
mirror-scsi0: switching to actively synced mode
mirror-scsi0: successfully switched to actively synced mode
2026-01-23 14:18:28 starting online/live migration on unix:/run/qemu-server/99998.migrate
2026-01-23 14:18:28 set migration capabilities
2026-01-23 14:18:28 migration downtime limit: 100 ms
2026-01-23 14:18:28 migration cachesize: 256.0 MiB
2026-01-23 14:18:28 set migration parameters
2026-01-23 14:18:28 start migrate command to unix:/run/qemu-server/99998.migrate
2026-01-23 14:18:29 migration active, transferred 56.5 MiB of 2.0 GiB VM-state, 78.1 MiB/s
2026-01-23 14:18:30 migration active, transferred 111.3 MiB of 2.0 GiB VM-state, 55.6 MiB/s
2026-01-23 14:18:31 migration active, transferred 160.4 MiB of 2.0 GiB VM-state, 1.6 GiB/s
2026-01-23 14:18:32 migration active, transferred 213.4 MiB of 2.0 GiB VM-state, 52.6 MiB/s
2026-01-23 14:18:33 migration active, transferred 266.1 MiB of 2.0 GiB VM-state, 63.7 MiB/s
2026-01-23 14:18:34 migration active, transferred 321.8 MiB of 2.0 GiB VM-state, 54.1 MiB/s
2026-01-23 14:18:35 migration active, transferred 375.8 MiB of 2.0 GiB VM-state, 67.5 MiB/s
2026-01-23 14:18:37 average migration speed: 229.4 MiB/s - downtime 94 ms
2026-01-23 14:18:37 migration completed, transferred 437.9 MiB VM-state
2026-01-23 14:18:37 migration status: completed
all 'mirror' jobs are ready
mirror-scsi0: Completing block job...
mirror-scsi0: Completed successfully.
mirror-scsi0: mirror-job finished
2026-01-23 14:18:39 stopping NBD storage migration server on target.
2026-01-23 14:18:41 stopping migration dbus-vmstate helpers
2026-01-23 14:18:41 migrated 0 conntrack state entries
2026-01-23 14:18:46 flushing conntrack state for guest on source node
2026-01-23 14:18:50 migration finished successfully (duration 00:01:57)
TASK OKFurther, life migration seems to be much faster. Might I make any weird mistake? I do not find anything helpful in the server log files. If I can provide anything that might help, please let me know.
Regards,
Fant
Last edited: