Hi all,
unfortunately, I'm unable to use new live migration with local disks. It always fails with different causes.
command used:
#qm migrate 5008 node2 --online --with-local-disks --migration_type insecure
One example:
drive-virtio0: transferred: 10926161920 bytes remaining: 10549133312 bytes total: 21475295232 bytes progression: 50.88 % busy: 1 ready: 0
drive-virtio2: transferred: 5369561088 bytes remaining: 0 bytes total: 5369561088 bytes progression: 100.00 % busy: 0 ready: 1
drive-virtio0: transferred: 11041505280 bytes remaining: 10433789952 bytes total: 21475295232 bytes progression: 51.41 % busy: 1 ready: 0
drive-virtio2: transferred: 5369561088 bytes remaining: 0 bytes total: 5369561088 bytes progression: 100.00 % busy: 0 ready: 1
drive-virtio0: transferred: 11157897216 bytes remaining: 10317398016 bytes total: 21475295232 bytes progression: 51.96 % busy: 1 ready: 0
drive-virtio2: Cancelling block job
drive-virtio0: Cancelling block job
drive-virtio2: Done.
drive-virtio0: Done.
2017-07-12 14:55:23 ERROR: online migrate failure - mirroring error: drive-virtio2: mirroring has been cancelled
2017-07-12 14:55:23 aborting phase 2 - cleanup resources
2017-07-12 14:55:23 migrate_cancel
2017-07-12 14:55:29 ERROR: command '/usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-3' failed: exit code 5
2017-07-12 14:55:34 ERROR: command '/usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-1' failed: exit code 5
2017-07-12 14:55:40 ERROR: command '/usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-2' failed: exit code 5
2017-07-12 14:55:42 ERROR: migration finished with problems (duration 00:03:11)
migration problems
And cleanup above fails , but works if running manually:
# /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-3
Logical volume "vm-5008-disk-3" successfully removed
# /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-2
Logical volume "vm-5008-disk-2" successfully removed
# /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-1
Logical volume "vm-5008-disk-1" successfully removed
Second failed example:
drive-virtio0: transferred: 21477261312 bytes remaining: 0 bytes total: 21477261312 bytes progression: 100.00 % busy: 0 ready: 1
drive-virtio2: transferred: 5372248064 bytes remaining: 0 bytes total: 5372248064 bytes progression: 100.00 % busy: 0 ready: 1
drive-virtio1: transferred: 5368709120 bytes remaining: 0 bytes total: 5368709120 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
2017-07-12 14:43:51 starting online/live migration on unix:/run/qemu-server/5008.migrate
2017-07-12 14:43:51 migrate_set_speed: 8589934592
2017-07-12 14:43:51 migrate_set_downtime: 0.1
2017-07-12 14:43:51 set migration_caps
2017-07-12 14:43:51 set cachesize: 429496729
2017-07-12 14:43:51 start migrate command to unix:/run/qemu-server/5008.migrate
2017-07-12 14:43:53 migration status: active (transferred 194938628, remaining 2529087488), total 4312604672)
2017-07-12 14:43:53 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:43:55 migration status: active (transferred 426817335, remaining 2294706176), total 4312604672)
2017-07-12 14:43:55 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:43:57 migration status: active (transferred 659232271, remaining 2062286848), total 4312604672)
2017-07-12 14:43:57 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:43:59 migration status: active (transferred 890355473, remaining 1828827136), total 4312604672)
2017-07-12 14:43:59 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:01 migration status: active (transferred 1122538713, remaining 1595625472), total 4312604672)
2017-07-12 14:44:01 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:03 migration status: active (transferred 1352968248, remaining 1357312000), total 4312604672)
2017-07-12 14:44:03 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:05 migration status: active (transferred 1575328780, remaining 586825728), total 4312604672)
2017-07-12 14:44:05 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:07 migration status: active (transferred 1806870933, remaining 350924800), total 4312604672)
2017-07-12 14:44:07 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:09 migration status: active (transferred 2039005014, remaining 114012160), total 4312604672)
2017-07-12 14:44:09 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:09 migration status: active (transferred 2074399569, remaining 77950976), total 4312604672)
2017-07-12 14:44:09 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:09 migration status: active (transferred 2109882801, remaining 42536960), total 4312604672)
2017-07-12 14:44:09 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
query migrate failed: VM 5008 not running
2017-07-12 14:44:10 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:11 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:12 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:13 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:15 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:16 query migrate failed: VM 5008 not running
2017-07-12 14:44:16 ERROR: online migrate failure - too many query migrate failures - aborting
2017-07-12 14:44:16 aborting phase 2 - cleanup resources
2017-07-12 14:44:16 migrate_cancel
2017-07-12 14:44:16 migrate_cancel error: VM 5008 not running
drive-virtio0: Cancelling block job
drive-virtio2: Cancelling block job
drive-virtio1: Cancelling block job
2017-07-12 14:44:16 ERROR: VM 5008 not running
2017-07-12 14:44:22 ERROR: migration finished with problems (duration 00:05:20)
migration problems
unfortunately, I'm unable to use new live migration with local disks. It always fails with different causes.
command used:
#qm migrate 5008 node2 --online --with-local-disks --migration_type insecure
One example:
drive-virtio0: transferred: 10926161920 bytes remaining: 10549133312 bytes total: 21475295232 bytes progression: 50.88 % busy: 1 ready: 0
drive-virtio2: transferred: 5369561088 bytes remaining: 0 bytes total: 5369561088 bytes progression: 100.00 % busy: 0 ready: 1
drive-virtio0: transferred: 11041505280 bytes remaining: 10433789952 bytes total: 21475295232 bytes progression: 51.41 % busy: 1 ready: 0
drive-virtio2: transferred: 5369561088 bytes remaining: 0 bytes total: 5369561088 bytes progression: 100.00 % busy: 0 ready: 1
drive-virtio0: transferred: 11157897216 bytes remaining: 10317398016 bytes total: 21475295232 bytes progression: 51.96 % busy: 1 ready: 0
drive-virtio2: Cancelling block job
drive-virtio0: Cancelling block job
drive-virtio2: Done.
drive-virtio0: Done.
2017-07-12 14:55:23 ERROR: online migrate failure - mirroring error: drive-virtio2: mirroring has been cancelled
2017-07-12 14:55:23 aborting phase 2 - cleanup resources
2017-07-12 14:55:23 migrate_cancel
2017-07-12 14:55:29 ERROR: command '/usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-3' failed: exit code 5
2017-07-12 14:55:34 ERROR: command '/usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-1' failed: exit code 5
2017-07-12 14:55:40 ERROR: command '/usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-2' failed: exit code 5
2017-07-12 14:55:42 ERROR: migration finished with problems (duration 00:03:11)
migration problems
And cleanup above fails , but works if running manually:
# /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-3
Logical volume "vm-5008-disk-3" successfully removed
# /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-2
Logical volume "vm-5008-disk-2" successfully removed
# /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=selma' root@10.1.11.6 pvesm free local-lvm:vm-5008-disk-1
Logical volume "vm-5008-disk-1" successfully removed
Second failed example:
drive-virtio0: transferred: 21477261312 bytes remaining: 0 bytes total: 21477261312 bytes progression: 100.00 % busy: 0 ready: 1
drive-virtio2: transferred: 5372248064 bytes remaining: 0 bytes total: 5372248064 bytes progression: 100.00 % busy: 0 ready: 1
drive-virtio1: transferred: 5368709120 bytes remaining: 0 bytes total: 5368709120 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
2017-07-12 14:43:51 starting online/live migration on unix:/run/qemu-server/5008.migrate
2017-07-12 14:43:51 migrate_set_speed: 8589934592
2017-07-12 14:43:51 migrate_set_downtime: 0.1
2017-07-12 14:43:51 set migration_caps
2017-07-12 14:43:51 set cachesize: 429496729
2017-07-12 14:43:51 start migrate command to unix:/run/qemu-server/5008.migrate
2017-07-12 14:43:53 migration status: active (transferred 194938628, remaining 2529087488), total 4312604672)
2017-07-12 14:43:53 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:43:55 migration status: active (transferred 426817335, remaining 2294706176), total 4312604672)
2017-07-12 14:43:55 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:43:57 migration status: active (transferred 659232271, remaining 2062286848), total 4312604672)
2017-07-12 14:43:57 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:43:59 migration status: active (transferred 890355473, remaining 1828827136), total 4312604672)
2017-07-12 14:43:59 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:01 migration status: active (transferred 1122538713, remaining 1595625472), total 4312604672)
2017-07-12 14:44:01 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:03 migration status: active (transferred 1352968248, remaining 1357312000), total 4312604672)
2017-07-12 14:44:03 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:05 migration status: active (transferred 1575328780, remaining 586825728), total 4312604672)
2017-07-12 14:44:05 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:07 migration status: active (transferred 1806870933, remaining 350924800), total 4312604672)
2017-07-12 14:44:07 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:09 migration status: active (transferred 2039005014, remaining 114012160), total 4312604672)
2017-07-12 14:44:09 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:09 migration status: active (transferred 2074399569, remaining 77950976), total 4312604672)
2017-07-12 14:44:09 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-12 14:44:09 migration status: active (transferred 2109882801, remaining 42536960), total 4312604672)
2017-07-12 14:44:09 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
query migrate failed: VM 5008 not running
2017-07-12 14:44:10 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:11 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:12 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:13 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:15 query migrate failed: VM 5008 not running
query migrate failed: VM 5008 not running
2017-07-12 14:44:16 query migrate failed: VM 5008 not running
2017-07-12 14:44:16 ERROR: online migrate failure - too many query migrate failures - aborting
2017-07-12 14:44:16 aborting phase 2 - cleanup resources
2017-07-12 14:44:16 migrate_cancel
2017-07-12 14:44:16 migrate_cancel error: VM 5008 not running
drive-virtio0: Cancelling block job
drive-virtio2: Cancelling block job
drive-virtio1: Cancelling block job
2017-07-12 14:44:16 ERROR: VM 5008 not running
2017-07-12 14:44:22 ERROR: migration finished with problems (duration 00:05:20)
migration problems