Hi all,
VM migration fails at an inconstant rate between nodes:
The VM just doesn't restart once migrated and this weird behaviour is not systematic. I've just mass migrated 4 VM between 2 nodes and only one of them failed, staying in a paused state.
The faulty VM starts again perfectly as soon as I starts it manually.
My nodes are all at the same version and up to date.
Does anyone have an idea of the possible reason for this kind of annoying behaviour?
Thanks.
VM migration fails at an inconstant rate between nodes:
Code:
2019-03-19 13:59:14 use dedicated network address for sending migration traffic (10.0.0.102)
2019-03-19 13:59:14 starting migration of VM 108 to node 'srv-pve2' (10.0.0.102)
2019-03-19 13:59:14 copying disk images
2019-03-19 13:59:14 starting VM 108 on remote node 'srv-pve2'
2019-03-19 13:59:16 start remote tunnel
2019-03-19 13:59:17 ssh tunnel ver 1
2019-03-19 13:59:17 starting online/live migration on unix:/run/qemu-server/108.migrate
2019-03-19 13:59:17 migrate_set_speed: 8589934592
2019-03-19 13:59:17 migrate_set_downtime: 0.1
2019-03-19 13:59:17 set migration_caps
2019-03-19 13:59:17 set cachesize: 1073741824
2019-03-19 13:59:17 start migrate command to unix:/run/qemu-server/108.migrate
2019-03-19 13:59:18 migration status: active (transferred 62487693, remaining 4696678400), total 8607571968)
2019-03-19 13:59:18 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:19 migration status: active (transferred 178044604, remaining 4517789696), total 8607571968)
2019-03-19 13:59:19 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:20 migration status: active (transferred 294914274, remaining 4362051584), total 8607571968)
2019-03-19 13:59:20 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:21 migration status: active (transferred 412347831, remaining 4244140032), total 8607571968)
2019-03-19 13:59:21 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:22 migration status: active (transferred 529399129, remaining 4126883840), total 8607571968)
2019-03-19 13:59:22 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:24 migration status: active (transferred 646372245, remaining 4007931904), total 8607571968)
2019-03-19 13:59:24 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:25 migration status: active (transferred 763947983, remaining 3890544640), total 8607571968)
2019-03-19 13:59:25 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:26 migration status: active (transferred 881253982, remaining 3765460992), total 8607571968)
2019-03-19 13:59:26 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:27 migration status: active (transferred 998329940, remaining 3648163840), total 8607571968)
2019-03-19 13:59:27 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:28 migration status: active (transferred 1112858764, remaining 3532746752), total 8607571968)
2019-03-19 13:59:28 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:29 migration status: active (transferred 1229844255, remaining 3415617536), total 8607571968)
2019-03-19 13:59:29 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:30 migration status: active (transferred 1347149435, remaining 3298361344), total 8607571968)
2019-03-19 13:59:30 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:31 migration status: active (transferred 1464454615, remaining 3181105152), total 8607571968)
2019-03-19 13:59:31 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:32 migration status: active (transferred 1581668958, remaining 3058597888), total 8607571968)
2019-03-19 13:59:32 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:33 migration status: active (transferred 1698973148, remaining 2921291776), total 8607571968)
2019-03-19 13:59:33 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:34 migration status: active (transferred 1816252912, remaining 2774601728), total 8607571968)
2019-03-19 13:59:34 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:35 migration status: active (transferred 1933938819, remaining 2642485248), total 8607571968)
2019-03-19 13:59:35 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:36 migration status: active (transferred 2050012025, remaining 2513764352), total 8607571968)
2019-03-19 13:59:36 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:37 migration status: active (transferred 2156409826, remaining 2375208960), total 8607571968)
2019-03-19 13:59:37 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:38 migration status: active (transferred 2273664390, remaining 2200875008), total 8607571968)
2019-03-19 13:59:38 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:39 migration status: active (transferred 2390863226, remaining 2053742592), total 8607571968)
2019-03-19 13:59:39 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:40 migration status: active (transferred 2508165490, remaining 1906130944), total 8607571968)
2019-03-19 13:59:40 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:41 migration status: active (transferred 2625466296, remaining 1779683328), total 8607571968)
2019-03-19 13:59:41 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:42 migration status: active (transferred 2724722472, remaining 1037291520), total 8607571968)
2019-03-19 13:59:42 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:43 migration status: active (transferred 2834117992, remaining 805498880), total 8607571968)
2019-03-19 13:59:43 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:44 migration status: active (transferred 2951536662, remaining 558317568), total 8607571968)
2019-03-19 13:59:44 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:45 migration status: active (transferred 3066632261, remaining 382504960), total 8607571968)
2019-03-19 13:59:45 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:46 migration status: active (transferred 3183743653, remaining 256532480), total 8607571968)
2019-03-19 13:59:46 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:47 migration status: active (transferred 3300974231, remaining 128503808), total 8607571968)
2019-03-19 13:59:47 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3411972418, remaining 405504), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3423808967, remaining 159277056), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 2827 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3435989691, remaining 147103744), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 5795 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3447879031, remaining 135221248), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 8692 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3460051511, remaining 123072512), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 11658 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3472184842, remaining 110100480), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 14613 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3483998140, remaining 93691904), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 17490 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3496281626, remaining 79478784), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 20482 overflow 0
2019-03-19 13:59:48 migration status: active (transferred 3508258213, remaining 39067648), total 8607571968)
2019-03-19 13:59:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 23385 overflow 0
2019-03-19 13:59:49 migration status: active (transferred 3520462282, remaining 25591808), total 8607571968)
2019-03-19 13:59:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 26358 overflow 0
2019-03-19 13:59:49 migration status: active (transferred 3532622873, remaining 13262848), total 8607571968)
2019-03-19 13:59:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 29321 overflow 0
2019-03-19 13:59:49 migration status: active (transferred 3544553352, remaining 12673024), total 8607571968)
2019-03-19 13:59:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 32228 overflow 0
2019-03-19 13:59:49 migration speed: 256.00 MB/s - downtime 134 ms
2019-03-19 13:59:49 migration status: completed
2019-03-19 13:59:49 ERROR: tunnel replied 'ERR: resume failed - unable to find configuration file for VM 108 - no such machine' to command 'resume 108'
2019-03-19 13:59:52 ERROR: migration finished with problems (duration 00:00:38)
TASK ERROR: migration problemsThe VM just doesn't restart once migrated and this weird behaviour is not systematic. I've just mass migrated 4 VM between 2 nodes and only one of them failed, staying in a paused state.
The faulty VM starts again perfectly as soon as I starts it manually.
My nodes are all at the same version and up to date.
Does anyone have an idea of the possible reason for this kind of annoying behaviour?
Thanks.