online migration failure

[dt47]

Hello,I was updating my Server and wanted to live migrate all the machines in the cluster, which went successful with all but one. There I got the following message:

Jul 09 13:41:53 starting migration of VM 134 to node 'franz' (127.0.0.9)
Jul 09 13:41:53 copying disk images
Jul 09 13:41:53 starting VM 134 on remote node 'franz'
Jul 09 13:41:54 starting ssh migration tunnel
Jul 09 13:41:54 starting online/live migration on localhost:60000
Jul 09 13:41:54 migrate_set_speed: 8589934592
Jul 09 13:41:54 migrate_set_downtime: 0.1
Jul 09 13:41:56 ERROR: online migrate failure - aborting
Jul 09 13:41:56 aborting phase 2 - cleanup resources
Jul 09 13:41:56 migrate_cancel
Jul 09 13:41:57 ERROR: migration finished with problems (duration 00:00:04)
migration problems

I hoped, doing it over the command line would produce something more verbose, but it did not.

pveversion -v

proxmox-ve-2.6.32: 3.2-129 (running kernel: 2.6.32-29-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-27-pve: 2.6.32-121
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-24-pve: 2.6.32-111
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-25-pve: 2.6.32-113
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-16
pve-firmware: 1.1-3
libpve-common-perl: 3.0-18
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1

The disks are shared over drbd.Do you have an idea how to troubleshoot this?edit: I do not really understand why the text isnt verbatim between the code-tags.. neither with code nor with CODE.bc the pasted text is unreadable like this I put it somewhere else too..: https://pastee.org/tq2za

 

[MasterTH]

please post the pveversion of both server and the hardware-specs of each node

 

[dt47]

Hi and thanks.The output of pveversion for the other node is:

proxmox-ve-2.6.32: 3.2-126 (running kernel: 2.6.32-29-pve)pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)pve-kernel-2.6.32-20-pve: 2.6.32-100pve-kernel-2.6.32-27-pve: 2.6.32-121pve-kernel-2.6.32-19-pve: 2.6.32-96pve-kernel-2.6.32-24-pve: 2.6.32-111pve-kernel-2.6.32-16-pve: 2.6.32-82pve-kernel-2.6.32-28-pve: 2.6.32-124pve-kernel-2.6.32-25-pve: 2.6.32-113pve-kernel-2.6.32-22-pve: 2.6.32-107pve-kernel-2.6.32-29-pve: 2.6.32-126pve-kernel-2.6.32-26-pve: 2.6.32-114pve-kernel-2.6.32-23-pve: 2.6.32-109lvm2: 2.02.98-pve4clvm: 2.02.98-pve4corosync-pve: 1.4.5-1openais-pve: 1.1.4-3libqb0: 0.11.1-2redhat-cluster-pve: 3.2.0-2resource-agents-pve: 3.9.2-4fence-agents-pve: 4.0.5-1pve-cluster: 3.0-12qemu-server: 3.1-16pve-firmware: 1.1-3libpve-common-perl: 3.0-18libpve-access-control: 3.0-11libpve-storage-perl: 3.0-19pve-libspice-server1: 0.12.4-3vncterm: 1.1-6vzctl: 4.0-1pve5vzprocps: 2.0.11-2vzquota: 3.1-2pve-qemu-kvm: 1.7-8ksm-control-daemon: 1.1-1glusterfs-client: 3.4.2-1

The Hardware is identical. It is a Dell PowerEdge R610 (http://www.dell.com/downloads/global/products/pedge/en/server-poweredge-r610-specs-en.pdf).The output of lspci is:

00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02)00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02)00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02)00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02)00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)00:1f.2 IDE interface: Intel Corporation 82801IB (ICH9) 2 port SATA Controller [IDE mode] (rev 02)01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)04:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)05:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)06:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)

SAN is also in use. Besides that there are two virtual disks (RAID1 and 5), one for the System with about 150 GB and one for the two DRBD volumes with 900 GB.e: It appears every new line I type in here is ignored (using the latest version of chromium) :/ Here are the outputs of pveversion -v and lspci again: https://pastee.org/xa3wr

edit2: I shut the VM down today and updated the cluster, so this thread will be of no help to anyone with a similiar problem now.