Live migration error - how to get more detailed log/debug

[teer]

Same problem here. I installed updated qemu-server deb on all nodes, but this has no effect.

 

[teer]

Hm... shutdowning/starting all VMs on one node makes live migration works again (for those VMs).

How I suppose to debug it? Seems like it is no migrations logs anywhere...

 

[wolfgang]

Debug is not that easy.
If your vm's has this error before the update applied, the update can't fix the state.

 

[teer]

How can I prevent/debug such situation in future? It is not first occurence of problem.

 

[srozanski]

How can I prevent/debug such situation in future? It is not first occurence of problem.

I agree with teer. It's not the first time. What we should do at start to begin reasonable debug research?

 

[teer]

Ok, next round. VM from next node doesn't migrate with same symptoms:

Feb 04 15:10:37 starting migration of VM 11048 to node 'zabdb' (192.168.110.14)
Feb 04 15:10:37 copying disk images
Feb 04 15:10:38 starting VM 11048 on remote node 'zabdb'
Feb 04 15:10:40 starting ssh migration tunnel
Feb 04 15:10:40 starting online/live migration on localhost:60000
Feb 04 15:10:40 migrate_set_speed: 8589934592
Feb 04 15:10:40 migrate_set_downtime: 0.1
Feb 04 15:10:42 ERROR: online migrate failure - aborting
Feb 04 15:10:42 aborting phase 2 - cleanup resources
Feb 04 15:10:42 migrate_cancel
Feb 04 15:10:44 ERROR: migration finished with problems (duration 00:00:07)
TASK ERROR: migration problems

both source and target nodes have similar packages (after yesterday qemu-server update):

root@rd350:~# pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-52
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1

How to resolve this problem? (I think I can reboot VM to "fix" live migrate, but it is definitely a kludge )

 

[teer]

If I understand QemuMigrate.pm/QemuServer.pm correctly, observed symptoms can only be observed when "failed"/"cancelled" has been read from VM's communication socket (/var/run/qemu-server/VMID.qmp). This is very strange, because I have no qemu-guest-agent in VM's and, moreover, no "Qemu agent" option checked on any VM. How it can be?

 

[teer]

I tried to migrate problem VM (only one on problem node) again and it freezes forever with such node log:

Feb 05 09:12:15 rd350 pmxcfs[12830]: [status] notice: received log
Feb 05 09:12:57 rd350 pvedaemon[7439]: <teer@pve> starting task UPID:rd350:0000324A:091D8B02:56B40529:qmigrate:11048:teer@pve:
Feb 05 09:12:58 rd350 pmxcfs[12830]: [status] notice: received log
Feb 05 09:12:59 rd350 pmxcfs[12830]: [status] notice: received log
Feb 05 09:13:04 rd350 pvedaemon[7439]: got timeout
Feb 05 09:13:14 rd350 pvedaemon[7439]: unable to connect to VM 11048 qmp socket - timeout after 31 retries
Feb 05 09:13:24 rd350 pvedaemon[7439]: unable to connect to VM 11048 qmp socket - timeout after 31 retries
Feb 05 09:13:33 rd350 pvedaemon[7439]: worker exit
Feb 05 09:13:33 rd350 pvedaemon[1945]: worker 7439 finished
Feb 05 09:13:33 rd350 pvedaemon[1945]: starting 1 worker(s)
Feb 05 09:13:33 rd350 pvedaemon[1945]: worker 12929 started
Feb 05 09:13:35 rd350 pvedaemon[10854]: unable to connect to VM 11048 qmp socket - timeout after 31 retries
Feb 05 09:13:45 rd350 pvedaemon[10854]: unable to connect to VM 11048 qmp socket - timeout after 31 retries
Feb 05 09:13:55 rd350 pvedaemon[10854]: unable to connect to VM 11048 qmp socket - timeout after 31 retries

VM is become irresponsible and I have to stop migration and restart VM. After that, it runs fine and live-migrated successfully.