KVM VM freeze after Live Migration

[Lokytech]

Hello everyone,

I've played with Proxmox since 4 months now and i got a problem with the live migration function.
I saw on KVM website that there is a problem with the tsc clocksource and after many research i'm stuck on this. My VM with only 1 vcpu can be live migrated without a problem but as soon as i have more than 1 vcpu, the VM freeze after live migration and i have to restart it.

Both Proxmox servers have the same hardware :

Dell PowerEdge 2950 with Intel(R) Xeon(R) CPU E5420 @ 2.50GHz Quad Core

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 6
cpu MHz		: 2493.989
cache size	: 6144 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips	: 4987.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

Here is my 2 Proxmox servers versions :

[root@med-virt-1 ~]# pveversion -v
pve-manager: 1.6-2 (pve-manager/1.6/5087)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.6-19
pve-kernel-2.6.32-3-pve: 2.6.32-14
pve-kernel-2.6.32-4-pve: 2.6.32-19
qemu-server: 1.1-18
pve-firmware: 1.0-8
libpve-storage-perl: 1.0-14
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-7
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.12.5-1
ksm-control-daemon: 1.0-4

[root@med-virt-2 ~]# pveversion -v
pve-manager: 1.6-2 (pve-manager/1.6/5087)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.6-19
pve-kernel-2.6.32-3-pve: 2.6.32-14
pve-kernel-2.6.32-4-pve: 2.6.32-19
qemu-server: 1.1-18
pve-firmware: 1.0-8
libpve-storage-perl: 1.0-14
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-7
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.12.5-1
ksm-control-daemon: 1.0-4

Here is the KVM VM conf (debian x86_64) :

cat 101.conf 
name: MED-FTP
ide2: ISO-MED-NAS:iso/clonezilla-live-1.2.6-20-amd64.iso,media=cdrom
vlan0: virtio=A6:6C:71:3A:E9:1B
bootdisk: virtio0
ostype: l26
memory: 1024
onboot: 1
sockets: 4
boot: c
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
cores: 1
description: 
virtio0: MED-NAS:101/vm-101-disk-1.raw

On the VM i've tried all those kernel boot option with no succes :
- acpi=off noapic
- clocksource=hpet nosmp noapic nolapic notsc
- clocksource=acpi_pm
- divider=10 clocksource=acpi_pm lpj=n
- divider=10 notsc lpj=n


I've attached the dmesg message of both proxmox servers and the debian vm.

I also tried to put the kernel boot option "processor.max_cstate=1" on the Proxmox server with also no success ...

If you have any idea of how i can have live migration working without freezing my VMs, i'm ready to test everything you ask me.

Thanks for your help.

 

[dietmar]

Does it work when you use ide instead of virtio?

 

[Lokytech]

I give it a try and come back to tell you.

 

[Lokytech]

Switch the hdd from virtio to ide remove the freeze for the moment.
But how is that possible ?

IDE is the slowest emulation, isn't it ?

Why VIRTIO is responsible for the freeze ?

I found this post from Zachary Amsden : http://amailbox.org/mailarchive/linux-kernel/2010/9/18/4621523
I thought it was exactly my problem. But if switching the hdd from VIRTIO to IDE solves the problem. I was totally wrong ...

Can i use SCSI safely or should i test ?

 

[dietmar]

IDE is the slowest emulation, isn't it ?

it is quite fast.

Why VIRTIO is responsible for the freeze ?

a bug.

Can i use SCSI safely or should i test ?

Why do you want to use SCSI?

 

[Lokytech]

Why do you want to use SCSI?

As i saw on this post :
http://arstechnica.com/civis/viewtopic.php?f=16&t=34039

I would like the best hdd IO performance.

In fact, i host on 3 clustered Proxmox servers :
- 2 Mysql server,
- 1 IIS 6,
- 1 ProFTPd linked to mysql,
- 1 .NET Server
And everyone work together. Around 5k/sec basic SQL queries on both MySQL server. So IO performance is quite important for me.

Anyway, Thanks a lot for this workaround.

Do you know the ID of this bug in order to follow it and switch back to VIRTIO when it's corrected ?

For the moment we use 1 win 2k3 storage server for NFS but in 2 weeks we will switch to 2 OpenFiler HA (DRBD, HeartBeat, vIP, iSCSI). Is there the VIRTIO bug on iSCSI storage or does it affect only NFS storage ?

---------
Edit : I switched from IDE to SCSI and Live Migration seems to work.

I tried on another VM but the CPU went up to 100% and froze. Solution : kernel parameter : clocksource=acpi_pm

 

[jamel]

hi lokytech

you mean that you can do the live migration between 2 VM using KVM ?

 

[Lokytech]

Hi Jamel,

No, i mean i can move a running VM between 2 physicals Proxmox servers.

 

[jamel]

hi

ok can I have a small summary about how you can do it and i hope that i have your email ( gmail ) the mine is jamel.mrabet@gmail.com

have you tried to make a migration with KVM/qemu ?


thx man

 

[Lokytech]

Proxmox is a bare-metal distribution. It's a Debian Linux with Openvz and KVM/Qemu with a graphical web interface to manage it.
Have a look to the wiki :
http://pve.proxmox.com/wiki/Category:Video_Tutorials
http://pve.proxmox.com/wiki/Installation