2.0RC1, Live Migration VM Crashes after a few minutes

M

mdevilz

New Member
Dec 9, 2011
29
0
1
Hi,

I am testing two clustered computers with a shared ISCSI LVM storage. I setup a 45GB VM and can successfully migrate between the two vm's however after a couple of minutes to about 15 minutes the VM just stops. I am not sure where to look for any logs, so any help would be appreciated.

Thanks!

Code: 
Output of pveversion -v

pve-manager: 2.0-33 (pve-manager/2.0/c598d9e1)
running kernel: 2.6.32-7-pve
proxmox-ve-2.6.32: 2.0-60
pve-kernel-2.6.32-7-pve: 2.6.32-60
lvm2: 2.02.88-2pve1
clvm: 2.02.88-2pve1
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-1
pve-cluster: 1.0-23
qemu-server: 2.0-19
pve-firmware: 1.0-15
libpve-common-perl: 1.0-14
libpve-access-control: 1.0-15
libpve-storage-perl: 2.0-12
vncterm: 1.0-2
vzctl: 3.0.30-2pve1
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-3
ksm-control-daemon: 1.1-1
 
Last edited:
Actually it seems to crash instantly after I press the console to open a new one, or when I press the reload button.

Here is what is on the log tab for the VM
Code: 
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:24 proton pvedaemon[1921]: <root@pam> starting task UPID:proton:00000C6D:0001D613:4F45B834:qmigrate:100:root@pam:[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:25 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:25 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:33 proton kernel: vmbr0: port 2(tap100i0) entering disabled state[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:33 proton kernel: vmbr0: port 2(tap100i0) entering disabled state[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:35 proton pvedaemon[1921]: <root@pam> end task UPID:proton:00000C6D:0001D613:4F45B834:qmigrate:100:root@pam: OK[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:36 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:36 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:38 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:53:38 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:54:35 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:54:35 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:54:45 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:54:47 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:54:59 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:02 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:02 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:03 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:05 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:08 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:09 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:09 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:24 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:25 proton qm[3286]: <root@pam> starting task UPID:proton:00000CD7:00020559:4F45B8AD:qmstart:100:root@pam:[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:25 proton qm[3287]: start VM 100: UPID:proton:00000CD7:00020559:4F45B8AD:qmstart:100:root@pam:[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:25 proton kernel: device tap100i0 entered promiscuous mode[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:25 proton kernel: vmbr0: port 2(tap100i0) entering forwarding state[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:25 proton kernel: New device tap100i0 does not support netpoll[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:25 proton kernel: Disabling netpoll for vmbr0[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:26 proton qm[3286]: <root@pam> end task UPID:proton:00000CD7:00020559:4F45B8AD:qmstart:100:root@pam: OK[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:32 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:32 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:34 proton pvedaemon[1923]: <root@pam> successful auth for user 'root@pam'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:34 proton kernel: vmbr0: port 2(tap100i0) entering disabled state[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:34 proton kernel: vmbr0: port 2(tap100i0) entering disabled state[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:34 proton pmxcfs[1512]: [status] notice: received log[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Feb 22 22:55:35 proton pmxcfs[1512]: [status] notice: received log
[/FONT][/COLOR]
 
It is a regular VM not a container.

Config below:
Code: 
bootdisk: virtio0
cores: 2
ide2: none,media=cdrom
memory: 2048
name: city17-p
net0: virtio=00:26:B9:43:9C:36,bridge=vmbr0
ostype: l26
sockets: 1
virtio0: iscsi-lvm:vm-100-disk-1
OS is Scientific Linux 6.1
 
I am having the same problem. The guest crashes after migration if you open the console. Only on certain OS's though.

It happens with these OS's

Fedora 16, 64 Bit. with latest updates installed. Tried VirtIO and IDE
Centos 6.2 64 Bit
OpenSUSE 12.1, 64-Bit

These ones don't have the problem

Windows 7 Professional, 32 Bit
Ubuntu 10.04, 64 Bit

That is all I have tested so far.

I installed 2.0 RC 1 yesterday from the ISO. At first I could not start KVM virtual machines. Running "aptitude update && aptitude full-upgrade" fixed that problem, but console problem remains.

Let me know if you need further info.
 
Debian 6 also doesn't seem to have this problem. Unfortunately, most of my VM's are Scientific Linux :(
 
mdevilz said:
Debian 6 also doesn't seem to have this problem. Unfortunately, most of my VM's are Scientific Linux :(
Click to expand...

I've also noticed that the mouse stops tracking properly in Windows 2008 R2 after live migration.
While it does not crash, something is clearly happening differently after a live migration is performed.

I wonder if disabling the mouse driver(or maybe all USB related drivers) in the kernel would allow live migration to work.
 
spirit said:
Hi, did you mouse problems in vnc console ? or also through tse/rdp ?
Click to expand...
In the VNC Console.

Edit:
I just discovered that the difference is what mouse is being used after live migration.
After live migration the mouse does not track properly, I open up monitor and run "mouse_set 1" and it now tracks properly.
"mouse_set 0" makes it track improperly.
So it appears that the mouse input gets switched from tablet mouse to normal mouse upon live migration.
Doubt this is causing OSs to crash, but is something worth testing.
 
Last edited:
e100 said:
In the VNC Console.

Edit:
I just discovered that the difference is what mouse is being used after live migration.
After live migration the mouse does not track properly, I open up monitor and run "mouse_set 1" and it now tracks properly.
"mouse_set 0" makes it track improperly.
So it appears that the mouse input gets switched from tablet mouse to normal mouse upon live migration.
Doubt this is causing OSs to crash, but is something worth testing.
Click to expand...

I found a msg in qemu mailing with same problem !

http://www.mail-archive.com/qemu-devel@nongnu.org/msg98298.html

can you do a "info usb" in monitor, before and after migration ?

patch here:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg81807.html
"qemu uses the ps/2 mouse by default. The usb tablet (or mouse) is
activated as soon as qemu sees some guest activity on the device,i.e. polling for HID events. That used to work fine for both freshboot and migration.Remote wakeup support changed the picture though: There will be nopolling after migration in case the guest suspended the usb bus,waiting for wakeup events. Result is that the ps/2 mouse staysactive.Fix this by activating the usb tablet / mouse in post_load() in casethe guest enabled remote wakeup."I'll compile a kvm package with package to test it.

UPDATE : Dawn, the patch is already applied to kvm source... Please post "info usb" to see if the problem is related...

UPDATE : another report here
http://www.mail-archive.com/kvm@vger.kernel.org/msg68006.html
 
Last edited:
renstar said:
I am having the same problem. The guest crashes after migration if you open the console. Only on certain OS's though.

It happens with these OS's

Fedora 16, 64 Bit. with latest updates installed. Tried VirtIO and IDE
Centos 6.2 64 Bit
OpenSUSE 12.1, 64-Bit

These ones don't have the problem

Windows 7 Professional, 32 Bit
Ubuntu 10.04, 64 Bit

That is all I have tested so far.

I installed 2.0 RC 1 yesterday from the ISO. At first I could not start KVM virtual machines. Running "aptitude update && aptitude full-upgrade" fixed that problem, but console problem remains.

Let me know if you need further info.
Click to expand...

Hi, for your linux vm, do you use Xorg server ? (and connect with vnc console ?)
 
I have tried both. With and without Xorg and both fail.

I installed the deb package and it still fails, it also causes this error if I try to start a VNC console to a guest on one server from the admin interface from the other.

"Error unable to find configuration file for VM 103 - no such machine (500)"
 
I have can confirm that the mouse pointer issue seems to be resolved. I can also VNC to guests on either host now. The crash remains, but I think you already know that. :)
 
Ok, fine, i have send the patch to proxmox dev !

I'll try to help you for your vm crash.

Some questions :

Does it crash only when you use console ? Or does it crash after sometime without doing nothing ?
Could you send your vms config file ? (/etc/pve/qemu-server/XXX.conf)
Could you send your vms kernel version ? (uname -a)
 
As far as I can tell it only crashes when I connect to the console (I will do some more tests to confirm). It connects briefly and then it shuts down the machine.

Code: 
root@vm2:~# more /etc/pve/qemu-server/102.conf
bootdisk: virtio0
cores: 1
ide2: none,media=cdrom
memory: 1024
name: Centos62
net0: virtio=C6:3E:8D:7E:4B:07,bridge=vmbr0
ostype: l26
sockets: 1
virtio0: shared:vm-102-disk-1

Code: 
root@vm2:~# uname -a
Linux vm2 2.6.32-7-pve #1 SMP Thu Feb 16 09:00:32 CET 2012 x86_64 GNU/Linux
 
I'm using shared iSCSI LVM storage.

Centos Kernel version.
Code: 
Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back