Online migration fails

[darkpixel]

I have three proxmox boxes (kvm1, kvm2, kvm3). All the virtual machines are running under kvm using iscsi targets as their storage.

It doesn't seem to matter what host I am transferring to or what guest I am moving--they fail:

kvm3:~# qmigrate --online --verbose 10.47.0.181 109
Mar 22 09:55:29 starting migration of VM 109 to host '10.47.0.181'
Mar 22 09:55:29 copying disk images
Mar 22 09:55:30 starting VM on remote host '10.47.0.181'
Mar 22 09:55:30 starting migration tunnel
Mar 22 09:55:30 starting online/live migration
Mar 22 09:55:32 migration status: failed
Mar 22 09:55:33 online migrate failure - aborting
Mar 22 09:55:33 migration finished with problems (duration 00:00:04)
kvm3:~#

Sometimes the VM is left in a running state, but pulling up the console displays:
serial0 console

Other times the VM is stopped.
Either way, I can immediately start the VM and it comes up without problems.

I see nothing unusual in syslog on either machine.
From the source machine (kvm3):

Mar 22 09:55:29 kvm3 qmigrate[19345]: starting migration of VM 109 to host '10.47.0.181'
Mar 22 09:55:29 kvm3 qmigrate[19345]: copying disk images
Mar 22 09:55:30 kvm3 qmigrate[19345]: starting VM on remote host '10.47.0.181'
Mar 22 09:55:30 kvm3 qmigrate[19345]: starting migration tunnel
Mar 22 09:55:30 kvm3 qmigrate[19345]: starting online/live migration
Mar 22 09:55:30 kvm3 qmigrate[19345]: VM 109 monitor command 'migrate -d "tcp:localhost:60000"'
Mar 22 09:55:32 kvm3 qmigrate[19345]: migration status: failed
Mar 22 09:55:32 kvm3 qmigrate[19345]: VM 109 stopping
Mar 22 09:55:32 kvm3 kernel: vmbr0: port 2(tap109i0d0) entering disabled state
Mar 22 09:55:32 kvm3 kernel: vmbr0: port 2(tap109i0d0) entering disabled state
Mar 22 09:55:32 kvm3 kernel: vmbr100: port 2(tap109i100d0) entering disabled state
Mar 22 09:55:32 kvm3 kernel: vmbr100: port 2(tap109i100d0) entering disabled state
Mar 22 09:55:32 kvm3 kernel: vmbr6: port 2(tap109i6d0) entering disabled state
Mar 22 09:55:32 kvm3 kernel: vmbr6: port 2(tap109i6d0) entering disabled state
Mar 22 09:55:33 kvm3 qmigrate[19345]: online migrate failure - aborting
Mar 22 09:55:33 kvm3 qmigrate[19345]: migration finished with problems (duration 00:00:04)


From the target machine (kvm1):
Mar 22 09:55:30 kvm1 qm[4053]: VM 109 start
Mar 22 09:55:30 kvm1 kernel: device tap109i0 entered promiscuous mode
Mar 22 09:55:30 kvm1 kernel: vmbr0: port 4(tap109i0) entering forwarding state
Mar 22 09:55:30 kvm1 kernel: device tap109i100 entered promiscuous mode
Mar 22 09:55:30 kvm1 kernel: vmbr100: port 2(tap109i100) entering forwarding state
Mar 22 09:55:30 kvm1 kernel: device tap109i6 entered promiscuous mode
Mar 22 09:55:30 kvm1 kernel: vmbr6: port 6(tap109i6) entering forwarding state


Any pointers?
I can easily move virtual machines around after-hours to free up two machines for testing if needed.

 

[dietmar]

What is the output of:

# pveversion -v

 

[spirit]

I don't know if it's related,but i have sometime this problem with direct iscsi lun in vm. (without lvm).

if the lun is not available in the target host (not discovered by exemple), proxmox doesn't verify it and vm crash during migration.

 

[darkpixel]

*facepalm* I had it in my clipboard, but forgot to paste it. Sorry.

kvm1:~# pveversion -v
pve-manager: 1.7-11 (pve-manager/1.7/5470)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.18: 1.7-10
pve-kernel-2.6.32-4-pve: 2.6.32-30
pve-kernel-2.6.18-2-pve: 2.6.18-5
pve-kernel-2.6.18-4-pve: 2.6.18-10
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm-2.6.18: 0.9.1-10

kvm2:~# pveversion -v
pve-manager: 1.7-11 (pve-manager/1.7/5470)
running kernel: 2.6.18-4-pve
proxmox-ve-2.6.18: 1.7-10
pve-kernel-2.6.32-4-pve: 2.6.32-30
pve-kernel-2.6.18-2-pve: 2.6.18-5
pve-kernel-2.6.18-4-pve: 2.6.18-10
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm-2.6.18: 0.9.1-10
kvm2:~#


kvm3:~# pveversion -v
pve-manager: 1.7-11 (pve-manager/1.7/5470)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.7-30
pve-kernel-2.6.32-4-pve: 2.6.32-30
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-3
ksm-control-daemon: 1.0-4
kvm3:~#


kvm2 has the latest kernel installed, we just haven't rebooted the thing in a while...

 

[tom]

*facepalm* I had it in my clipboard, but forgot to paste it. Sorry.

kvm1:~# pveversion -v
pve-manager: 1.7-11 (pve-manager/1.7/5470)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.18: 1.7-10
pve-kernel-2.6.32-4-pve: 2.6.32-30
pve-kernel-2.6.18-2-pve: 2.6.18-5
pve-kernel-2.6.18-4-pve: 2.6.18-10
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm-2.6.18: 0.9.1-10

unsupported mix of packages. you run the 2.6.32 kernel but you installed the KVM package for the 2.6.18 kernel.