[SOLVED] ERROR: online migrate failure - unable to parse migration status 'setup'

F

Florent

Member
Apr 3, 2012
91
2
8
Hi,

I have a cluster with up-to-date nodes, but not all have been restarted after upgrade (I can't for the moment...).

When I try to migrate a VM from one host to another, online migration process fails with :

Code: 
ERROR: online migrate failure - unable to parse migration status 'setup' - aborting

Of course, I can do off-line migration, or reboot my nodes. But in a HA environment, how do we have to do ?

I searched in sources, and I couldn't find "setup" status in migration process. The condition that fails is (in QemuMigrate.pm) :

Code: 
if ($stat->{status} =~ m/^(active|completed|failed|cancelled)$/im) {

How can a node send "setup" status if it's not handled here ? I repeat, all nodes are up-to-date (aptitude update; aptitude safe-upgrade), so the Perl code is up-to-date.

Can a developer give me information about that "setup" status and how to handle that ?

Thank you.

Florent
 
Last edited:
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

What is the output ov

# pveversion -v
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

I want to migrate from host6 to host5 :

host5 is new (installed yesterday), and host6 is old (103 days uptime).

Code: 
root@host6:~# pveversion -v
proxmox-ve-2.6.32: 3.2-126 (running kernel: 2.6.32-26-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-3.10.0-2-pve: 3.10.0-10
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-16
pve-firmware: 1.1-3
libpve-common-perl: 3.0-18
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1

Code: 
root@host5:~# pveversion -v
proxmox-ve-2.6.32: 3.2-126 (running kernel: 3.10.0-2-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-3.10.0-2-pve: 3.10.0-10
pve-kernel-2.6.32-29-pve: 2.6.32-126
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-16
pve-firmware: 1.1-3
libpve-common-perl: 3.0-18
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

Any information ?
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

does it work if you run 2.6.32 kernel on both nodes?
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

I would suggest rebooting the node you have not rebooted yet after upgrading. I did not get the same error but i had issues with migrating VM in same scenario. I do not know how many VMs you are dealing and how many proxmox node you got. But it wont be impossible to migrate all VMs from non restarted node to another node. Will save you lot of hassle and unexpected behavior due to not restarting.
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

tom said:
does it work if you run 2.6.32 kernel on both nodes?
Click to expand...

It does not seem related to kernel version. I have the same issue with two 3.10 nodes (both fresh installed, with a new VM) or with two 2.6.32 nodes.

This issue is not appearing on a local test cluster in our offices (all 3.10) but only appears in our online cluster.

spirit said:
Hi,
http://www.redhat.com/archives/libvir-list/2013-July/msg01765.html

seem to be a new state since last qemu version
Click to expand...

Well done. This status does not seem to be supported yet by Proxmox (my first message).

Maybe in our local cluster, the "setup" state is fast and PVE does not see it. But in our online cluster, this state take more time and PVE does not recognize it ?

Can a Proxmox team member can confirm this ?

symmcom said:
I would suggest rebooting the node you have not rebooted yet after upgrading. I did not get the same error but i had issues with migrating VM in same scenario. I do not know how many VMs you are dealing and how many proxmox node you got. But it wont be impossible to migrate all VMs from non restarted node to another node. Will save you lot of hassle and unexpected behavior due to not restarting.
Click to expand...

Yes of course it could a solution as I said. But maybe it could be better to find the origin of this trouble.
 
Last edited:
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

Florent said:
Maybe in our local cluster, the "setup" state is fast and PVE does not see it. But in our online cluster, this state take more time and PVE does not recognize it ?

Can a Proxmox team member can confirm this ?
Click to expand...

Yes, It's quite possible. "setup" is a new state in qemu 1.7. It's mainly usage is for infiniband migration.
I can't reproduce it here, but can you try to insert in

/usr/share/perl5/PVE/QemuMigrate.pm


Code: 
        if ($stat->{status} =~ m/^(setup)$/im) {
            sleep(1);
            next;
        }
        if ($stat->{status} =~ m/^(active|completed|failed|cancelled)$/im) {

then
/etc/init.d/pvedaemon restart
/etc/init.d/pveproxy restart


and test migration again
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

It seems to loop in "setup" state... what could cause that ?
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

Florent said:
It seems to loop in "setup" state... what could cause that ?
Click to expand...

it's possible that migration don't start, maybe network problem.
is the ssh tunnel between both nodes is correctly open ?
also, before launching migration, check that on target host, you don't have a phantom kvm process with
# ps -aux| /usr/bin/kvm -id vmid
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

spirit said:
it's possible that migration don't start, maybe network problem.
is the ssh tunnel between both nodes is correctly open ?
also, before launching migration, check that on target host, you don't have a phantom kvm process with
# ps -aux| /usr/bin/kvm -id vmid
Click to expand...

Status of migration stays at :

Code: 
May 26 15:25:43 starting migration of VM 141 to node 'host5' (10.111.0.5)
May 26 15:25:43 copying disk images
May 26 15:25:43 starting VM 141 on remote node 'host5'
May 26 15:25:48 starting ssh migration tunnel
May 26 15:25:49 starting online/live migration on localhost:60000
May 26 15:25:49 migrate_set_speed: 8589934592
May 26 15:25:49 migrate_set_downtime: 0.1

So tunnel is done, no ?

Of course I can ssh from one node to the other. It does not seem to be a network problem..
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

do you spice ? I see some related code in qemu like

if (migration_in_setup(s)) {
spice_server_migrate_start(spice_server);
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

Here is VM Config :

Code: 
balloon: 2048
bootdisk: virtio0
cores: 2
ide2: none,media=cdrom
memory: 4096
name: maria02-2.priv.coppint.net
net0: virtio=F2:7D:30:29:47:79,bridge=vmbr10
onboot: 1
ostype: l26
sockets: 1
virtio0: PVE01-RBD01:vm-141-disk-1,cache=writeback,size=64G

Not using Spice...
 
Re: ERROR: online migrate failure - unable to parse migration status 'setup' - aborti

Hi everyone,

I have found the cause of this problem : ip6tables rules on my nodes.

Before, I was DROPPING all IPv6 :

Code: 
ip6tables -P INPUT DROP
ip6tables -P FORWARD DROP
ip6tables -P OUTPUT DROP

Now, I only DROP IPv6 incoming to my public interface (vmbr1, bridging eth0) :


Code: 
ip6tables -A INPUT ! -i vmbr1 -j ACCEPT
ip6tables -P INPUT DROP
ip6tables -P FORWARD ACCEPT
ip6tables -P OUTPUT ACCEPT

Nothing was changed in v4 rules.

IPv6 is never configured in nodes, they all use IPv4, public or private network.

This is very nice but I'm trying to understand why :)

Does anyone have an idea ? :eek:

Thank you.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back