cannot move vms from one node to another

[raqdedicados]

Hi there

We have a problem with one of our cluster servers

WE can move vms along without problem except from one cluster that hangs migrating to another cluster or master any vm... it keep on displaying this message and nothing happen...

/usr/bin/ssh -t -t -n -o BatchMode=yes XXXXXXXXXXX /usr/sbin/vzmigrate XXXXXXXXXXX 113
Starting migration of CT 113 to XXXXXXXXXXX
Preparing remote node
Initializing remote quota
Syncing private

All the nodes are running version 1.8 and have identical hardware and local storage; with the other cluster node we have no problems moving vms along....

Any idea wht this could be happen or from wich point i can start looking ?

Thanks

 

[tom]

post the output of 'pveversion -v ' of all involved cluster nodes. do you migrate online or offline?

 

[raqdedicados]

we try offline and also online but same result

Hi ! here the node that is holding the vms we want to move

raq105:~# pveversion -v
pve-manager: 1.8-18 (pve-manager/1.8/6070)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.28-1pve1
vzdump: 1.2-14
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.1-1
ksm-control-daemon: 1.0-6

the other. from this cluster we can migrate to another without problem
raq100:~# pveversion -v
pve-manager: 1.8-18 (pve-manager/1.8/6070)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.28-1pve1
vzdump: 1.2-14
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.1-1
ksm-control-daemon: 1.0-6

 

[udo]

we try offline and also online but same result

Hi ! here the node that is holding the vms we want to move

raq105:~# pveversion -v
pve-manager: 1.8-18 (pve-manager/1.8/6070)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.28-1pve1
vzdump: 1.2-14
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.1-1
ksm-control-daemon: 1.0-6

the other. from this cluster we can migrate to another without problem
raq100:~# pveversion -v
pve-manager: 1.8-18 (pve-manager/1.8/6070)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.28-1pve1
vzdump: 1.2-14
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.1-1
ksm-control-daemon: 1.0-6

Hi,
and the config of the destination?

What's about storage - something strange with that on the target-node?

Any logs?

Udo

 

[raqdedicados]

mmm i think the strange thing must be on the source node, coz in other nodes there are no problems of migration offline or online
the message log does not say nothing weird the vzdump log also.... do u know where i can find more info ?

 

[raqdedicados]

something weird

i can move from raq100 to raq105 but i can t move from this to anyone else node. so all the vms are stuck there

Hi,
and the config of the destination?

What's about storage - something strange with that on the target-node?

Any logs?

Udo

 

[udo]

something weird

i can move from raq100 to raq105 but i can t move from this to anyone else node. so all the vms are stuck there

Hi,
look if the network-connection is ok:
on raq105 "iperf -s" and on raq100 "iperf -c raq105". After that vice versa (iperf -s on raq105).

BTW. why do you think that the error is in the code? When the migration on other hosts works (and it's works for many people).

Udo

 

[raqdedicados]

raq100:~# iperf -c raq105
connect failed: Connection refused
write1 failed: Broken pipe
write2 failed: Broken pipe
------------------------------------------------------------
Client connecting to raq105, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 0.0.0.0 port 41136 connected with 200.49.152.105 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec

and

raq105:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------

 

[udo]

raq100:~# iperf -c raq105
connect failed: Connection refused
write1 failed: Broken pipe
write2 failed: Broken pipe
------------------------------------------------------------
Client connecting to raq105, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 0.0.0.0 port 41136 connected with 200.49.152.105 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec

and

raq105:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------

Hi,
you must first start the listener (iperf -s) before start the client (iperf -c IP.ADD.RES.S ).
If raq105 don't be resolved use the ip-address instead.

Edit: i see, connection is ok but no traffic. Do you have an firewall between the nodes? It's looks so.

Udo

 

[raqdedicados]

here source

raq105:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 200.49.152.105 port 5001 connected with 200.49.152.100 port 47367
^CWaiting for server threads to complete. Interrupt again to force quit.
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 543 MBytes 456 Mbits/sec

here destination



raq100:~# iperf -c raq105
------------------------------------------------------------
Client connecting to raq105, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 200.49.152.100 port 47367 connected with 200.49.152.105 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 543 MBytes 456 Mbits/sec

on the web interface of proxmox ..still says-..

Running Maintainance Tasks

https://200.49.152.102/vmlist/index.htm?action=stop&ptoken=4e497a2c4215ff45199a

	migrate	Mon Aug 15 16:55:22 2011	root	13	113

CommandStart timeUserCIDVMID

in detail
usr/bin/ssh -t -t -n -o BatchMode=yes 200.49.152.105 /usr/sbin/vzmigrate 200.49.152.100 113
Starting migration of CT 113 to 200.49.152.100
Prep.....

and the only thing i can do is abort it. if we do it. the vm is on both node. but not operational in the destination