Migration Problem

Discussion in 'Proxmox VE: Installation and configuration' started by Pourya Mehdinejad, May 17, 2018.

  1. Pourya Mehdinejad

    Proxmox VE Subscriber

    Joined:
    Nov 8, 2017
    Messages:
    38
    Likes Received:
    0
    Hi guys
    Today I've found out that I need to update my nodes and reboot them. In order to prevent downtime in my production servers, I was live migrating VMs one by one to other nodes in my cluster.

    and migration problem occurred while migrating two of my VMs. They got rebooted and we experienced 2 minutes of downtime.
    I've just found this in error log as "migration problem"

    2018-05-17 16:55:54 migration status: completed
    2018-05-17 16:55:56 ERROR: tunnel replied 'ERR: resume failed - VM 130 not running' to command 'resume 130'
    2018-05-17 16:56:24 ERROR: migration finished with problems (duration 00:02:42)
    TASK ERROR: migration problems


    Any Idea ?
     
  2. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    878
    Likes Received:
    86
    It looks like the VM "died" during migration, that could be because you tried to migrate from a updated node to one running older versions, we and projects we use (e.e. QEMU/KVM) only guarantee that migrating from old to new works, vice versa can work but sometimes you may run into problems, that's not really avoidable, I'm afraid. Thus please either migrate before starting the update (preferred) or update all, then migrate one node free, reboot it, and do the same with the next node.

    Could you please post the full task log, should be available through the WebUI.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Pourya Mehdinejad

    Proxmox VE Subscriber

    Joined:
    Nov 8, 2017
    Messages:
    38
    Likes Received:
    0

    Thank you for your response
    I was migrating the VMs before I attempt to update and all of my nodes were running on the same version. I was going to free one node to upgrade it.
    It happens completely randomly, VMs suddenly dies at the end of migration.
    There is another interesting thing in the log is the average migration speed was "32.25 MB/s" which is quite slow, all of my servers are connected to each other and to the storage via 10Gbps Fiber and the migration speed is usually more than 100 MB/s. Could this be because of network load or something like that ?
    Here is the full task log


    task started by HA resource agent
    2018-05-17 16:53:42 starting migration of VM 130 to node 'master3' (172.27.3.12)
    2018-05-17 16:53:42 copying disk images
    2018-05-17 16:53:42 starting VM 130 on remote node 'master3'
    2018-05-17 16:53:46 start remote tunnel
    2018-05-17 16:53:47 ssh tunnel ver 1
    2018-05-17 16:53:47 starting online/live migration on unix:/run/qemu-server/130.migrate
    2018-05-17 16:53:47 migrate_set_speed: 8589934592
    2018-05-17 16:53:47 migrate_set_downtime: 0.1
    2018-05-17 16:53:47 set migration_caps
    2018-05-17 16:53:47 set cachesize: 536870912
    2018-05-17 16:53:47 start migrate command to unix:/run/qemu-server/130.migrate
    2018-05-17 16:53:48 migration status: active (transferred 70251553, remaining 4220518400), total 4312604672)
    2018-05-17 16:53:48 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:49 migration status: active (transferred 130150487, remaining 4110000128), total 4312604672)
    2018-05-17 16:53:49 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:50 migration status: active (transferred 195151431, remaining 3984736256), total 4312604672)
    2018-05-17 16:53:50 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:51 migration status: active (transferred 285214946, remaining 3830980608), total 4312604672)
    2018-05-17 16:53:51 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:52 migration status: active (transferred 327360025, remaining 3777282048), total 4312604672)
    2018-05-17 16:53:52 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:53 migration status: active (transferred 383475298, remaining 3702120448), total 4312604672)
    2018-05-17 16:53:53 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:54 migration status: active (transferred 462977098, remaining 3597139968), total 4312604672)
    2018-05-17 16:53:54 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:55 migration status: active (transferred 546651622, remaining 3482877952), total 4312604672)
    2018-05-17 16:53:55 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:56 migration status: active (transferred 586763781, remaining 3428134912), total 4312604672)
    2018-05-17 16:53:56 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:57 migration status: active (transferred 612489946, remaining 3390275584), total 4312604672)
    2018-05-17 16:53:57 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:58 migration status: active (transferred 627171258, remaining 3369476096), total 4312604672)
    2018-05-17 16:53:58 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:53:59 migration status: active (transferred 641823908, remaining 3348664320), total 4312604672)
    2018-05-17 16:53:59 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:00 migration status: active (transferred 658287701, remaining 3325460480), total 4312604672)
    2018-05-17 16:54:00 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:01 migration status: active (transferred 667197056, remaining 3313057792), total 4312604672)
    2018-05-17 16:54:01 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:02 migration status: active (transferred 673169155, remaining 3304894464), total 4312604672)
    2018-05-17 16:54:02 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:03 migration status: active (transferred 677712019, remaining 3298627584), total 4312604672)
    2018-05-17 16:54:03 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:04 migration status: active (transferred 683967384, remaining 3290140672), total 4312604672)
    2018-05-17 16:54:04 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:05 migration status: active (transferred 687232555, remaining 3285757952), total 4312604672)
    2018-05-17 16:54:05 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:06 migration status: active (transferred 708720511, remaining 3258998784), total 4312604672)
    2018-05-17 16:54:06 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:07 migration status: active (transferred 712365923, remaining 3254886400), total 4312604672)
    2018-05-17 16:54:07 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:08 migration status: active (transferred 723285027, remaining 3242876928), total 4312604672)
    2018-05-17 16:54:08 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:09 migration status: active (transferred 725763977, remaining 3240345600), total 4312604672)
    2018-05-17 16:54:09 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:10 migration status: active (transferred 732270536, remaining 3231215616), total 4312604672)
    2018-05-17 16:54:10 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:11 migration status: active (transferred 743724150, remaining 3214495744), total 4312604672)
    2018-05-17 16:54:11 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:12 migration status: active (transferred 877832963, remaining 2924929024), total 4312604672)
    2018-05-17 16:54:12 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:13 migration status: active (transferred 1048388534, remaining 2677669888), total 4312604672)
    2018-05-17 16:54:13 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:14 migration status: active (transferred 1259560438, remaining 2440622080), total 4312604672)
    2018-05-17 16:54:14 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:15 migration status: active (transferred 1444577398, remaining 2194333696), total 4312604672)
    2018-05-17 16:54:15 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:16 migration status: active (transferred 1486246530, remaining 2142896128), total 4312604672)
    2018-05-17 16:54:16 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:17 migration status: active (transferred 1499816916, remaining 2126336000), total 4312604672)
    2018-05-17 16:54:17 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:18 migration status: active (transferred 1512522448, remaining 2112004096), total 4312604672)
    2018-05-17 16:54:18 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:19 migration status: active (transferred 1514690025, remaining 2109542400), total 4312604672)
    2018-05-17 16:54:19 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:20 migration status: active (transferred 1521968323, remaining 2101420032), total 4312604672)
    2018-05-17 16:54:20 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:21 migration status: active (transferred 1523807670, remaining 2099245056), total 4312604672)
    2018-05-17 16:54:21 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:22 migration status: active (transferred 1561382106, remaining 2055122944), total 4312604672)
    2018-05-17 16:54:22 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:23 migration status: active (transferred 1563094247, remaining 2053066752), total 4312604672)
    2018-05-17 16:54:23 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:24 migration status: active (transferred 1566584022, remaining 2048966656), total 4312604672)
    2018-05-17 16:54:24 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:25 migration status: active (transferred 1593810627, remaining 2017783808), total 4312604672)
    2018-05-17 16:54:25 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:26 migration status: active (transferred 1630273851, remaining 1974329344), total 4312604672)
    2018-05-17 16:54:26 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:27 migration status: active (transferred 1716014048, remaining 1835016192), total 4312604672)
    2018-05-17 16:54:27 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:28 migration status: active (transferred 1765662749, remaining 1752616960), total 4312604672)
    2018-05-17 16:54:28 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:29 migration status: active (transferred 1792265673, remaining 1714540544), total 4312604672)
    2018-05-17 16:54:29 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:30 migration status: active (transferred 1820413149, remaining 1674268672), total 4312604672)
    2018-05-17 16:54:30 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:31 migration status: active (transferred 1846775160, remaining 1630289920), total 4312604672)
    2018-05-17 16:54:31 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:32 migration status: active (transferred 1882103753, remaining 1585127424), total 4312604672)
    2018-05-17 16:54:32 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:33 migration status: active (transferred 1907472595, remaining 1551462400), total 4312604672)
    2018-05-17 16:54:33 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:34 migration status: active (transferred 1925938576, remaining 1526505472), total 4312604672)
    2018-05-17 16:54:34 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:35 migration status: active (transferred 1958884815, remaining 1484636160), total 4312604672)
    2018-05-17 16:54:35 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:36 migration status: active (transferred 2091218856, remaining 1315086336), total 4312604672)
    2018-05-17 16:54:36 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:37 migration status: active (transferred 2112500323, remaining 1287262208), total 4312604672)
    2018-05-17 16:54:37 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:38 migration status: active (transferred 2130432980, remaining 1264619520), total 4312604672)
    2018-05-17 16:54:38 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:39 migration status: active (transferred 2170953086, remaining 1214296064), total 4312604672)
    2018-05-17 16:54:39 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:40 migration status: active (transferred 2212496238, remaining 1168027648), total 4312604672)
    2018-05-17 16:54:40 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:41 migration status: active (transferred 2267206225, remaining 1096900608), total 4312604672)
    2018-05-17 16:54:41 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:42 migration status: active (transferred 2310800982, remaining 1039450112), total 4312604672)
    2018-05-17 16:54:42 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:43 migration status: active (transferred 2389254792, remaining 949207040), total 4312604672)
    2018-05-17 16:54:43 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:44 migration status: active (transferred 2502446323, remaining 777039872), total 4312604672)
    2018-05-17 16:54:44 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:45 migration status: active (transferred 2600391286, remaining 629514240), total 4312604672)
    2018-05-17 16:54:45 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:46 migration status: active (transferred 2667984546, remaining 530231296), total 4312604672)
    2018-05-17 16:54:46 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:47 migration status: active (transferred 2672379506, remaining 524177408), total 4312604672)
    2018-05-17 16:54:47 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:48 migration status: active (transferred 2687765310, remaining 503279616), total 4312604672)
    2018-05-17 16:54:48 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:49 migration status: active (transferred 2699879128, remaining 486146048), total 4312604672)
    2018-05-17 16:54:49 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:50 migration status: active (transferred 2704340022, remaining 479903744), total 4312604672)
    2018-05-17 16:54:50 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:51 migration status: active (transferred 2710259510, remaining 471445504), total 4312604672)
    2018-05-17 16:54:51 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:52 migration status: active (transferred 2716039165, remaining 463261696), total 4312604672)
    2018-05-17 16:54:52 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:53 migration status: active (transferred 2722196217, remaining 454778880), total 4312604672)
    2018-05-17 16:54:53 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:54 migration status: active (transferred 2725272605, remaining 450580480), total 4312604672)
    2018-05-17 16:54:54 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:55 migration status: active (transferred 2729906200, remaining 444014592), total 4312604672)
    2018-05-17 16:54:55 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:56 migration status: active (transferred 2731372749, remaining 441909248), total 4312604672)
    2018-05-17 16:54:56 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:57 migration status: active (transferred 2732728247, remaining 440025088), total 4312604672)
    2018-05-17 16:54:57 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:58 migration status: active (transferred 2737201723, remaining 433647616), total 4312604672)
    2018-05-17 16:54:58 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:54:59 migration status: active (transferred 2739945831, remaining 429715456), total 4312604672)
    2018-05-17 16:54:59 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:00 migration status: active (transferred 2744419055, remaining 423452672), total 4312604672)
    2018-05-17 16:55:00 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:01 migration status: active (transferred 2749179496, remaining 416931840), total 4312604672)
    2018-05-17 16:55:01 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:02 migration status: active (transferred 2752170005, remaining 412684288), total 4312604672)
    2018-05-17 16:55:02 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:03 migration status: active (transferred 2753755732, remaining 410386432), total 4312604672)
    2018-05-17 16:55:03 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:04 migration status: active (transferred 2755086633, remaining 408514560), total 4312604672)
    2018-05-17 16:55:04 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:05 migration status: active (transferred 2765585795, remaining 395673600), total 4312604672)
    2018-05-17 16:55:05 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:06 migration status: active (transferred 2768350063, remaining 391884800), total 4312604672)
    2018-05-17 16:55:06 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:07 migration status: active (transferred 2777633182, remaining 378961920), total 4312604672)
    2018-05-17 16:55:07 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:08 migration status: active (transferred 2783347046, remaining 370905088), total 4312604672)
    2018-05-17 16:55:08 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:09 migration status: active (transferred 2786682102, remaining 366399488), total 4312604672)
    2018-05-17 16:55:09 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:10 migration status: active (transferred 2794148734, remaining 356429824), total 4312604672)
    2018-05-17 16:55:10 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:11 migration status: active (transferred 2800129745, remaining 347930624), total 4312604672)
    2018-05-17 16:55:11 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:12 migration status: active (transferred 2804878207, remaining 341270528), total 4312604672)
    2018-05-17 16:55:12 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:13 migration status: active (transferred 2809437227, remaining 335101952), total 4312604672)
    2018-05-17 16:55:13 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:14 migration status: active (transferred 2817162950, remaining 324632576), total 4312604672)
    2018-05-17 16:55:14 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:15 migration status: active (transferred 2823037068, remaining 316325888), total 4312604672)
    2018-05-17 16:55:15 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:16 migration status: active (transferred 2829206314, remaining 307888128), total 4312604672)
    2018-05-17 16:55:16 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:17 migration status: active (transferred 2841134945, remaining 291180544), total 4312604672)
    2018-05-17 16:55:17 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:19 migration status: active (transferred 2850442606, remaining 278274048), total 4312604672)
    2018-05-17 16:55:19 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:20 migration status: active (transferred 2854878713, remaining 272134144), total 4312604672)
    2018-05-17 16:55:20 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:21 migration status: active (transferred 2858008722, remaining 267763712), total 4312604672)
    2018-05-17 16:55:21 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:22 migration status: active (transferred 2863767793, remaining 259633152), total 4312604672)
    2018-05-17 16:55:22 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:23 migration status: active (transferred 2872640449, remaining 247148544), total 4312604672)
    2018-05-17 16:55:23 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:24 migration status: active (transferred 2877248781, remaining 240898048), total 4312604672)
    2018-05-17 16:55:24 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:25 migration status: active (transferred 2882021542, remaining 234364928), total 4312604672)
    2018-05-17 16:55:25 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:26 migration status: active (transferred 2886285660, remaining 228220928), total 4312604672)
    2018-05-17 16:55:26 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:27 migration status: active (transferred 2898246991, remaining 211529728), total 4312604672)
    2018-05-17 16:55:27 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:28 migration status: active (transferred 2901319643, remaining 207171584), total 4312604672)
    2018-05-17 16:55:28 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:29 migration status: active (transferred 2906108648, remaining 200704000), total 4312604672)
    2018-05-17 16:55:29 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:30 migration status: active (transferred 2909045661, remaining 196575232), total 4312604672)
    2018-05-17 16:55:30 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:31 migration status: active (transferred 2912146771, remaining 192311296), total 4312604672)
    2018-05-17 16:55:31 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:32 migration status: active (transferred 2915190165, remaining 188219392), total 4312604672)
    2018-05-17 16:55:32 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:33 migration status: active (transferred 2921347062, remaining 179814400), total 4312604672)
    2018-05-17 16:55:33 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:34 migration status: active (transferred 2927413203, remaining 171712512), total 4312604672)
    2018-05-17 16:55:34 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:35 migration status: active (transferred 2930641708, remaining 167243776), total 4312604672)
    2018-05-17 16:55:35 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:36 migration status: active (transferred 2939567959, remaining 154595328), total 4312604672)
    2018-05-17 16:55:36 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:37 migration status: active (transferred 2941227783, remaining 152121344), total 4312604672)
    2018-05-17 16:55:37 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:38 migration status: active (transferred 2950079748, remaining 139735040), total 4312604672)
    2018-05-17 16:55:38 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:39 migration status: active (transferred 2957456821, remaining 129523712), total 4312604672)
    2018-05-17 16:55:39 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:40 migration status: active (transferred 2963753183, remaining 121008128), total 4312604672)
    2018-05-17 16:55:40 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:41 migration status: active (transferred 2966727195, remaining 116813824), total 4312604672)
    2018-05-17 16:55:41 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:42 migration status: active (transferred 2971196323, remaining 110555136), total 4312604672)
    2018-05-17 16:55:42 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:43 migration status: active (transferred 2980130108, remaining 98201600), total 4312604672)
    2018-05-17 16:55:43 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:44 migration status: active (transferred 2983244152, remaining 93638656), total 4312604672)
    2018-05-17 16:55:44 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:45 migration status: active (transferred 2986164525, remaining 89624576), total 4312604672)
    2018-05-17 16:55:45 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:46 migration status: active (transferred 2997924939, remaining 73056256), total 4312604672)
    2018-05-17 16:55:46 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:47 migration status: active (transferred 3001079582, remaining 68653056), total 4312604672)
    2018-05-17 16:55:47 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:48 migration status: active (transferred 3005388978, remaining 62406656), total 4312604672)
    2018-05-17 16:55:48 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:49 migration status: active (transferred 3009874550, remaining 56115200), total 4312604672)
    2018-05-17 16:55:49 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:50 migration status: active (transferred 3016002414, remaining 47874048), total 4312604672)
    2018-05-17 16:55:50 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:51 migration status: active (transferred 3022237252, remaining 39403520), total 4312604672)
    2018-05-17 16:55:51 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-05-17 16:55:52 migration status: active (transferred 3110872417, remaining 370089984), total 4312604672)
    2018-05-17 16:55:52 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 17356 overflow 0
    2018-05-17 16:55:53 migration status: active (transferred 3316878676, remaining 141357056), total 4312604672)
    2018-05-17 16:55:53 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 67540 overflow 0
    2018-05-17 16:55:54 migration speed: 32.25 MB/s - downtime 31 ms
    2018-05-17 16:55:54 migration status: completed
    2018-05-17 16:55:56 ERROR: tunnel replied 'ERR: resume failed - VM 130 not running' to command 'resume 130'
    2018-05-17 16:56:24 ERROR: migration finished with problems (duration 00:02:42)
    TASK ERROR: migration problems
     
  4. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    878
    Likes Received:
    86
    OK, you have done everything correctly.

    Hmm, looks weird... That the migration was so slow is strange (network congestions), but should really not lead to a dead VM on finish...
    Can you also please post your pveversion -v output, or tell me at leas which kernel and qemu version you used at the time this happened, those are the normal possible culprits and I could try to reproduce this...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. Pourya Mehdinejad

    Proxmox VE Subscriber

    Joined:
    Nov 8, 2017
    Messages:
    38
    Likes Received:
    0
    This is the output of pveversion -v now, I don't know about before update

    proxmox-ve: 5.2-2 (running kernel: 4.15.17-1-pve)
    pve-manager: 5.2-1 (running version: 5.2-1/0fcd7879)
    pve-kernel-4.15: 5.2-1
    pve-kernel-4.13: 5.1-44
    pve-kernel-4.15.17-1-pve: 4.15.17-9
    pve-kernel-4.13.16-2-pve: 4.13.16-48
    pve-kernel-4.13.13-6-pve: 4.13.13-42
    pve-kernel-4.13.13-5-pve: 4.13.13-38
    pve-kernel-4.13.13-4-pve: 4.13.13-35
    pve-kernel-4.13.4-1-pve: 4.13.4-26
    corosync: 2.4.2-pve5
    criu: 2.11.1-1~bpo90
    glusterfs-client: 3.8.8-1
    ksm-control-daemon: 1.2-2
    libjs-extjs: 6.0.1-2
    libpve-access-control: 5.0-8
    libpve-apiclient-perl: 2.0-4
    libpve-common-perl: 5.0-31
    libpve-guest-common-perl: 2.0-16
    libpve-http-server-perl: 2.0-8
    libpve-storage-perl: 5.0-23
    libqb0: 1.0.1-1
    lvm2: 2.02.168-pve6
    lxc-pve: 3.0.0-3
    lxcfs: 3.0.0-1
    novnc-pve: 0.6-4
    proxmox-widget-toolkit: 1.0-18
    pve-cluster: 5.0-27
    pve-container: 2.0-23
    pve-docs: 5.2-4
    pve-firewall: 3.0-8
    pve-firmware: 2.0-4
    pve-ha-manager: 2.0-5
    pve-i18n: 1.0-5
    pve-libspice-server1: 0.12.8-3
    pve-qemu-kvm: 2.11.1-5
    pve-xtermjs: 1.0-5
    qemu-server: 5.0-26
    smartmontools: 6.5+svn4324-1
    spiceterm: 3.0-5
    vncterm: 1.5-3
    zfsutils-linux: 0.7.8-pve1~bpo9
     
  6. Pourya Mehdinejad

    Proxmox VE Subscriber

    Joined:
    Nov 8, 2017
    Messages:
    38
    Likes Received:
    0
    Ok guys Today I was live migrating the VMs again, out of each 10 VMs, 7 of them ended up dead with the following error.
    Our Live migration seems not be useful anymore.

    2018-06-19 17:29:04 migration status: active (transferred 5753698414, remaining 89804800), total 8607571968)
    2018-06-19 17:29:04 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
    2018-06-19 17:29:05 migration status: active (transferred 5860837536, remaining 243789824), total 8607571968)
    2018-06-19 17:29:05 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 15929 overflow 0
    2018-06-19 17:29:06 migration speed: 115.38 MB/s - downtime 336 ms
    2018-06-19 17:29:06 migration status: completed
    2018-06-19 17:29:06 ERROR: tunnel replied 'ERR: resume failed - VM 146 not running' to command 'resume 146'
    2018-06-19 17:29:10 ERROR: migration finished with problems (duration 00:01:22)
    TASK ERROR: migration problems
     
  7. janos

    janos Member

    Joined:
    Aug 24, 2017
    Messages:
    113
    Likes Received:
    11
    Hello,

    We expect the same, but only from older version to newer version. We installed a new node its up to date, and i can not migrate to this node from older nodes.

    New node:
    Code:
    proxmox-ve: 5.2-2 (running kernel: 4.15.17-1-pve)
    pve-manager: 5.2-3 (running version: 5.2-3/785ba980)
    pve-kernel-4.15: 5.2-3
    pve-kernel-4.15.17-3-pve: 4.15.17-13
    pve-kernel-4.15.17-1-pve: 4.15.17-8
    corosync: 2.4.2-pve5
    criu: 2.11.1-1~bpo90
    glusterfs-client: 3.8.8-1
    ksm-control-daemon: not correctly installed
    libjs-extjs: 6.0.1-2
    libpve-access-control: 5.0-8
    libpve-apiclient-perl: 2.0-4
    libpve-common-perl: 5.0-34
    libpve-guest-common-perl: 2.0-17
    libpve-http-server-perl: 2.0-9
    libpve-storage-perl: 5.0-23
    libqb0: 1.0.1-1
    lvm2: 2.02.168-pve6
    lxc-pve: 3.0.0-3
    lxcfs: 3.0.0-1
    novnc-pve: 1.0.0-1
    openvswitch-switch: 2.7.0-2
    proxmox-widget-toolkit: 1.0-19
    pve-cluster: 5.0-27
    pve-container: 2.0-23
    pve-docs: 5.2-4
    pve-firewall: 3.0-12
    pve-firmware: 2.0-4
    pve-ha-manager: 2.0-5
    pve-i18n: 1.0-6
    pve-libspice-server1: 0.12.8-3
    pve-qemu-kvm: 2.11.1-5
    pve-xtermjs: 1.0-5
    qemu-server: 5.0-29
    smartmontools: 6.5+svn4324-1
    spiceterm: 3.0-5
    vncterm: 1.5-3
    
    
    Older node:
    Code:
    proxmox-ve: 5.2-2 (running kernel: 4.15.17-1-pve)
    pve-manager: 5.2-1 (running version: 5.2-1/0fcd7879)
    pve-kernel-4.15: 5.2-1
    pve-kernel-4.15.17-1-pve: 4.15.17-9
    pve-kernel-4.15.15-1-pve: 4.15.15-6
    corosync: 2.4.2-pve5
    criu: 2.11.1-1~bpo90
    glusterfs-client: 3.8.8-1
    ksm-control-daemon: not correctly installed
    libjs-extjs: 6.0.1-2
    libpve-access-control: 5.0-8
    libpve-apiclient-perl: 2.0-4
    libpve-common-perl: 5.0-31
    libpve-guest-common-perl: 2.0-16
    libpve-http-server-perl: 2.0-8
    libpve-storage-perl: 5.0-23
    libqb0: 1.0.1-1
    lvm2: 2.02.168-pve6
    lxc-pve: 3.0.0-3
    lxcfs: 3.0.0-1
    novnc-pve: 0.6-4
    openvswitch-switch: 2.7.0-2
    proxmox-widget-toolkit: 1.0-18
    pve-cluster: 5.0-27
    pve-container: 2.0-23
    pve-docs: 5.2-4
    pve-firewall: 3.0-8
    pve-firmware: 2.0-4
    pve-ha-manager: 2.0-5
    pve-i18n: 1.0-5
    pve-libspice-server1: 0.12.8-3
    pve-qemu-kvm: 2.11.1-5
    pve-xtermjs: 1.0-5
    qemu-server: 5.0-26
    smartmontools: 6.5+svn4324-1
    spiceterm: 3.0-5
    vncterm: 1.5-3
    xm8 ~ #
    
    
    The strange is, if i start VM on the new node, migrate back to the old one, and after i can migrate it to the new without issue.
     
  8. Kristin Chakarov

    Kristin Chakarov New Member

    Joined:
    Jul 7, 2018
    Messages:
    1
    Likes Received:
    0
    Hello everyone,

    We have encountered the same issue. We have a cluster of 8 nodes and several days ago a new node was installed and added to the cluster.

    versions output from older node which was updated:


    proxmox-ve: 5.2-2 (running kernel: 4.15.17-3-pve)
    pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
    pve-kernel-4.15: 5.2-3
    pve-kernel-4.15.17-3-pve: 4.15.17-14
    pve-kernel-4.13.13-5-pve: 4.13.13-38
    pve-kernel-4.13.4-1-pve: 4.13.4-26
    corosync: 2.4.2-pve5
    criu: 2.11.1-1~bpo90
    glusterfs-client: 3.8.8-1
    ksm-control-daemon: 1.2-2
    libjs-extjs: 6.0.1-2
    libpve-access-control: 5.0-8
    libpve-apiclient-perl: 2.0-5
    libpve-common-perl: 5.0-35
    libpve-guest-common-perl: 2.0-17
    libpve-http-server-perl: 2.0-9
    libpve-storage-perl: 5.0-23
    libqb0: 1.0.1-1
    lvm2: 2.02.168-pve6
    lxc-pve: 3.0.0-3
    lxcfs: 3.0.0-1
    novnc-pve: 1.0.0-1
    proxmox-widget-toolkit: 1.0-19
    pve-cluster: 5.0-27
    pve-container: 2.0-24
    pve-docs: 5.2-4
    pve-firewall: 3.0-12
    pve-firmware: 2.0-4
    pve-ha-manager: 2.0-5
    pve-i18n: 1.0-6
    pve-libspice-server1: 0.12.8-3
    pve-qemu-kvm: 2.11.1-5
    pve-xtermjs: 1.0-5
    qemu-server: 5.0-29
    smartmontools: 6.5+svn4324-1
    spiceterm: 3.0-5
    vncterm: 1.5-3
    zfsutils-linux: 0.7.9-pve1~bpo9

    versions output from the newly installed node:


    proxmox-ve: 5.2-2 (running kernel: 4.15.17-3-pve)
    pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
    pve-kernel-4.15: 5.2-3
    pve-kernel-4.15.17-3-pve: 4.15.17-14
    pve-kernel-4.13.13-2-pve: 4.13.13-33
    corosync: 2.4.2-pve5
    criu: 2.11.1-1~bpo90
    glusterfs-client: 3.8.8-1
    ksm-control-daemon: 1.2-2
    libjs-extjs: 6.0.1-2
    libpve-access-control: 5.0-8
    libpve-apiclient-perl: 2.0-5
    libpve-common-perl: 5.0-35
    libpve-guest-common-perl: 2.0-17
    libpve-http-server-perl: 2.0-9
    libpve-storage-perl: 5.0-23
    libqb0: 1.0.1-1
    lvm2: 2.02.168-pve6
    lxc-pve: 3.0.0-3
    lxcfs: 3.0.0-1
    novnc-pve: 1.0.0-1
    proxmox-widget-toolkit: 1.0-19
    pve-cluster: 5.0-27
    pve-container: 2.0-24
    pve-docs: 5.2-4
    pve-firewall: 3.0-12
    pve-firmware: 2.0-4
    pve-ha-manager: 2.0-5
    pve-i18n: 1.0-6
    pve-libspice-server1: 0.12.8-3
    pve-qemu-kvm: 2.11.1-5
    pve-xtermjs: 1.0-5
    qemu-server: 5.0-29
    smartmontools: 6.5+svn4324-1
    spiceterm: 3.0-5
    vncterm: 1.5-3
    zfsutils-linux: 0.7.9-pve1~bpo9
     
  9. Pourya Mehdinejad

    Proxmox VE Subscriber

    Joined:
    Nov 8, 2017
    Messages:
    38
    Likes Received:
    0
    It's completely random, I couldn't find any pattern when this issue might happen
     
  10. janos

    janos Member

    Joined:
    Aug 24, 2017
    Messages:
    113
    Likes Received:
    11
    Now i updated one node to the latest version, i tried to migrate from older node to the upgraded, and i got the same error. But i not see the reason, or any solution or even a workaround.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice