Live migration error - how to get more detailed log/debug

S

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
Hi,

After last cluster update, live migration is broken:

Jan 29 15:15:37 starting migration of VM 118 to node 'shv3' (10.64.2.3)
Jan 29 15:15:37 copying disk images
Jan 29 15:15:37 starting VM 118 on remote node 'shv3'
Jan 29 15:15:38 starting ssh migration tunnel
Jan 29 15:15:39 starting online/live migration on localhost:60000
Jan 29 15:15:39 migrate_set_speed: 8589934592
Jan 29 15:15:39 migrate_set_downtime: 0.1
Jan 29 15:15:41 ERROR: online migrate failure - aborting
Jan 29 15:15:41 aborting phase 2 - cleanup resources
Jan 29 15:15:41 migrate_cancel
Jan 29 15:15:41 ERROR: migration finished with problems (duration 00:00:04)
TASK ERROR: migration problems
Click to expand...

Where can I find more detailed log or turn on debug for this event? Why live migration isn't working as it should?
 
Hi,

is Ha enabled on your Cluster and on this VM?
 
Hi,

Ha is disabled :/ There is one small detail about this problem. I restarted only one node from three after apt-get upgrade command. On this two not updated is old kernel. But migration don't work between all nodes. So i have to turn off all VM on this two nodes which wasn't restart, and migrate them to the restarted one. And then turn VM on :/

- how to solve problem of live migration after upgrade, and
- how to get more detailed log about this event, in which file (/var/log) or which tool i should use (maybe some kvm tool)...

Any idea?
 
Last edited:
can you send the config of the vm
 
cat /etc/pve/qemu-server/118.conf
bootdisk: ide0
cores: 1
ide0: nfs-vmimg:1001/base-1001-disk-1.qcow2/118/vm-118-disk-1.qcow2,size=32G
memory: 1024
name: mint
net0: e1000=32:63:61:34:31:63,bridge=vmbr0,tag=203
numa: 0
ostype: l26
smbios1: uuid=f8ed631c-d59a-441d-9ac9-18f1d3a3da5f
sockets: 1
Click to expand...
 
I work on that and inform you if we have a solution.
 
Additional information's from nodes:

What shows pveversion command?
Installed modules status or maybe run modules status?​

Information from command "pveversion -v". Three nodes cluster (last was reset after upgrade):

- node 1:
pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.3-2-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2
Click to expand...

- node 2:
pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.3-2-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2
Click to expand...

- node 3:
pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2
Click to expand...
 
After reboot all nodes live migration works without any problems... But the problem is that I had to make offline migration to restart some nodes on production environment! :/
 
We have provide a patch in pvetest

qemu-server_4.0-52_amd64.deb
what will fix this bug.
 
What do you mean when you write "pvetest"? It is a repo?
How can i Install it? I don't see this update in proxmox update web panel after push upgrade button. Similar situation when I'm using "apt-get update; apt-get upgrade" command. List of new packages is equal 0?
 
This is what i have in my apt sources:

cat /etc/apt/sources.list
deb http://ftp.pl.debian.org/debian jessie main contrib

# Repozytorium rozwojowe
#deb http://download.proxmox.com/debian jessie pve-no-subscription

# security updates
deb http://security.debian.org jessie/updates main contrib

#deb http://http.us.debian.org/debian jessie main non-free
Click to expand...

cat /etc/apt/sources.list.d/pve-install-repo.list
deb http://download.proxmox.com/debian jessie pve-no-subscription
Click to expand...

wolfgang said:
http://download.proxmox.com/debian/dists/jessie/pvetest/binary-amd64/
You add this repo to your list
Click to expand...

How should above url look in apt sources file style?
 
If you get an new qemu-server packet form the pve-no-subscription this patch will be included.
 
wolfgang said:
If you get an new qemu-server packet form the pve-no-subscription this patch will be included.
Click to expand...

Ok, but the question is... When I install next release of proxmox on all nodes and restart only the first one, could I migrate all VMs from not restarted nodes to restarted one without this error? Or this patch works only with this update release?
 
This update need no restart at all and is compatible.
But install it on all nodes in the cluster.
I think what you mean ist the pve-qemu-kvm packed what has to be the same version to migrate.
 
wolfgang said:
This update need no restart at all and is compatible.
But install it on all nodes in the cluster.
I think what you mean ist the pve-qemu-kvm packed what has to be the same version to migrate.
Click to expand...


I just want to know what happend when... I will Install this patch, and everything would be ok with live migration in current version. After some time i will install new version of proxmox for ex. 4.9. Will be possibile live migration between:

node 1 upgraded and restarted
node 2 upgraded but not restarted (so some new modules can be different than in node 1)

After last upgrade i couldn't proceed live migration between node1 and node2 with different state of activate modules. Will this patch solve this problem in the future?
 
Normally there are no mig problems from oder version to newer ones.
 
Thank you very much for help to solve the first part of my question ;) Last part is:

- how to get more detailed information about live migration process, in which file (/var/log) or which tool i should use (maybe some qemu/kvm tool for testing live migration)...
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back