Live migration error - how to get more detailed log/debug

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
Hi,

After last cluster update, live migration is broken:

Jan 29 15:15:37 starting migration of VM 118 to node 'shv3' (10.64.2.3)
Jan 29 15:15:37 copying disk images
Jan 29 15:15:37 starting VM 118 on remote node 'shv3'
Jan 29 15:15:38 starting ssh migration tunnel
Jan 29 15:15:39 starting online/live migration on localhost:60000
Jan 29 15:15:39 migrate_set_speed: 8589934592
Jan 29 15:15:39 migrate_set_downtime: 0.1
Jan 29 15:15:41 ERROR: online migrate failure - aborting
Jan 29 15:15:41 aborting phase 2 - cleanup resources
Jan 29 15:15:41 migrate_cancel
Jan 29 15:15:41 ERROR: migration finished with problems (duration 00:00:04)
TASK ERROR: migration problems

Where can I find more detailed log or turn on debug for this event? Why live migration isn't working as it should?
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
498
103
Hi,

is Ha enabled on your Cluster and on this VM?
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
Hi,

Ha is disabled :/ There is one small detail about this problem. I restarted only one node from three after apt-get upgrade command. On this two not updated is old kernel. But migration don't work between all nodes. So i have to turn off all VM on this two nodes which wasn't restart, and migrate them to the restarted one. And then turn VM on :/

- how to solve problem of live migration after upgrade, and
- how to get more detailed log about this event, in which file (/var/log) or which tool i should use (maybe some kvm tool)...

Any idea?
 
Last edited:

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
498
103
can you send the config of the vm
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
cat /etc/pve/qemu-server/118.conf
bootdisk: ide0
cores: 1
ide0: nfs-vmimg:1001/base-1001-disk-1.qcow2/118/vm-118-disk-1.qcow2,size=32G
memory: 1024
name: mint
net0: e1000=32:63:61:34:31:63,bridge=vmbr0,tag=203
numa: 0
ostype: l26
smbios1: uuid=f8ed631c-d59a-441d-9ac9-18f1d3a3da5f
sockets: 1
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
498
103
I work on that and inform you if we have a solution.
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
Additional information's from nodes:

What shows pveversion command?
Installed modules status or maybe run modules status?​

Information from command "pveversion -v". Three nodes cluster (last was reset after upgrade):

- node 1:
pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.3-2-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2

- node 2:
pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.3-2-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2

- node 3:
pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
After reboot all nodes live migration works without any problems... But the problem is that I had to make offline migration to restart some nodes on production environment! :/
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
498
103
We have provide a patch in pvetest

qemu-server_4.0-52_amd64.deb
what will fix this bug.
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
What do you mean when you write "pvetest"? It is a repo?
How can i Install it? I don't see this update in proxmox update web panel after push upgrade button. Similar situation when I'm using "apt-get update; apt-get upgrade" command. List of new packages is equal 0?
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
This is what i have in my apt sources:

cat /etc/apt/sources.list
deb http://ftp.pl.debian.org/debian jessie main contrib

# Repozytorium rozwojowe
#deb http://download.proxmox.com/debian jessie pve-no-subscription

# security updates
deb http://security.debian.org jessie/updates main contrib

#deb http://http.us.debian.org/debian jessie main non-free

cat /etc/apt/sources.list.d/pve-install-repo.list
deb http://download.proxmox.com/debian jessie pve-no-subscription


How should above url look in apt sources file style?
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
498
103
If you get an new qemu-server packet form the pve-no-subscription this patch will be included.
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
If you get an new qemu-server packet form the pve-no-subscription this patch will be included.

Ok, but the question is... When I install next release of proxmox on all nodes and restart only the first one, could I migrate all VMs from not restarted nodes to restarted one without this error? Or this patch works only with this update release?
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
498
103
This update need no restart at all and is compatible.
But install it on all nodes in the cluster.
I think what you mean ist the pve-qemu-kvm packed what has to be the same version to migrate.
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
This update need no restart at all and is compatible.
But install it on all nodes in the cluster.
I think what you mean ist the pve-qemu-kvm packed what has to be the same version to migrate.


I just want to know what happend when... I will Install this patch, and everything would be ok with live migration in current version. After some time i will install new version of proxmox for ex. 4.9. Will be possibile live migration between:

node 1 upgraded and restarted
node 2 upgraded but not restarted (so some new modules can be different than in node 1)

After last upgrade i couldn't proceed live migration between node1 and node2 with different state of activate modules. Will this patch solve this problem in the future?
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
498
103
Normally there are no mig problems from oder version to newer ones.
 

srozanski

New Member
Nov 5, 2015
18
0
1
Poland
Thank you very much for help to solve the first part of my question ;) Last part is:

- how to get more detailed information about live migration process, in which file (/var/log) or which tool i should use (maybe some qemu/kvm tool for testing live migration)...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!