Live Migration Issue

[pitt]

Hello,
I set up proxmox 1.9 on Debian lenny with DRBD.
Everything is working fine, except live migration with a virtio Ethernet device.

It works using an e1000 network card. And it even works if i configure a virtio Ethernet device but don't install the driver in Windows (my test machine is a Windows 2008R2 installation). I tried different Ethernet virtio drivers which are around the net. Especially driver ISO from Fedora (http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/) and the WHQL certified version from Redhat (tried verison 1.2.0 and 1.3.3).

I suspect this is a KVM bug, but I'm not sure so I ask here first. I get following output in syslog on unsuccessful migration:
Source node:
Oct 3 16:41:09 pve1 pvedaemon[7033]: migrating VM 101 from node 1 (localhost) to node 2 (192.168.7.202)
Oct 3 16:41:09 pve1 qmigrate[7034]: starting migration of VM 101 to host '192.168.7.202'
Oct 3 16:41:09 pve1 qmigrate[7034]: copying disk images
Oct 3 16:41:09 pve1 qmigrate[7034]: starting VM on remote host '192.168.7.202'
Oct 3 16:41:10 pve1 qmigrate[7034]: starting migration tunnel
Oct 3 16:41:10 pve1 qmigrate[7034]: starting online/live migration
Oct 3 16:41:10 pve1 qmigrate[7034]: VM 101 monitor command 'migrate -d "tcp:localhost:60000"'
Oct 3 16:41:10 pve1 kernel: vmbr0: port 2(tap101i0d0) entering disabled state
Oct 3 16:41:10 pve1 kernel: vmbr0: port 2(tap101i0d0) entering disabled state
Oct 3 16:41:12 pve1 qmigrate[7034]: VM 101 monitor command failed - VM not running
Oct 3 16:41:12 pve1 qmigrate[7034]: VM 101 already stopped
Oct 3 16:41:12 pve1 qmigrate[7034]: online migrate failure - VM not running
Oct 3 16:41:12 pve1 qmigrate[7034]: migration finished with problems (duration 00:00:03)
Oct 3 16:41:12 pve1 pvedaemon[7033]: VM 101 migration failed -

Target node:
Oct 3 16:41:10 pve2 qm[7707]: VM 101 start
Oct 3 16:41:10 pve2 kernel: device tap101i0d0 entered promiscuous mode
Oct 3 16:41:10 pve2 kernel: vmbr0: port 2(tap101i0d0) entering forwarding state
Oct 3 16:41:11 pve2 kernel: vmbr0: port 2(tap101i0d0) entering disabled state
Oct 3 16:41:11 pve2 kernel: vmbr0: port 2(tap101i0d0) entering disabled state

PVE Version is:
pve1:~# pveversion -v
pve-manager: 1.9-24 (pve-manager/1.9/6542)
running kernel: 2.6.32-6-pve
proxmox-ve-2.6.32: 1.9-47
pve-kernel-2.6.32-6-pve: 2.6.32-47
qemu-server: 1.1-32
pve-firmware: 1.0-14
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-2pve1
vzdump: 1.2-16
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-1
ksm-control-daemon: 1.0-6
pve1:~#

After failed migration the VM is registered on the target node but "stopped".

I hope anybody has a hint where to investigate further. I'd really like to use the virtio driver because networking performance is much better compared to e1000.

best regards,
Christoph

 

[dietmar]

After failed migration the VM is registered on the target node but "stopped".

Any you can start it there?

 

[pitt]

Any you can start it there?

yes.

The VM runs on both machines flawless. If I do an offline migration there's absolutely no problem.
And the online migration works perfectly with an e1000 network card.

 

[dietmar]

Do you have the same CPU on both hosts (just wonder)?

 

[pitt]

Do you have the same CPU on both hosts (just wonder)?

Yes. Both machines are identical.
HP DL385G2, 12GB RAM, Smart Array Controller (with 5 146GB SAS disks) with battery backed write cache.
2x Dual-Core AMD Opteron(tm) Processor 2216 HE

 

[pitt]

I just tested live migration with an Ubuntu 10.04.3 LTS VM with the same result:
Live migration works with an emulated e1000 NIC and fails with virtio NIC.

Server Configfile e1000 NIC:
name: Test-Ubuntu
ide2: none,media=cdrom
bootdisk: virtio0
virtio0: drbd:vm-102-disk-1
ostype: l26
memory: 1024
sockets: 2
boot: c
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
vlan0: e1000=2A:F1:13:E9:72:38

Server Configfile virtio NIC:
name: Test-Ubuntu
ide2: none,media=cdrom
bootdisk: virtio0
virtio0: drbd:vm-102-disk-1
ostype: l26
memory: 1024
sockets: 2
boot: c
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
vlan0: virtio=5A:B8:46:4B:AF:98

 

[pitt]

This is very strange... It works now.
But I did not change anything. At least I think so .
I tried to do a migration "by hand" to get a better idea which command fails and suddenly it worked. I rebootet both machines and it still succeeds with virtio Ethernet NICs...
Whatever?