Live Migration Issues

[Luca Malaspina]

Hi,

Whenever we try to Live Migrate a VM from one of our clustered servers to another, it usually works no problem.

There is one VM in particular that does not migrate. It gives the following error message:


2018-03-29 06:34:42 starting migration of VM 113 to node 'proxbarw04' (10.10.100.153)
2018-03-29 06:34:42 copying disk images
2018-03-29 06:34:42 starting VM 113 on remote node 'proxbarw04'
2018-03-29 06:34:43 can't activate LV '/dev/vg_iscsi/vm-113-disk-1': device-mapper: create ioctl on vg_iscsi-vm--113--disk--1LVM-XdbNHpt4a5lA2TynjdGdbkqINFf08EEQKigfTHLT6v1blfhqtUTqe4BFUx10RI0T failed: Device or resource busy
2018-03-29 06:34:43 ERROR: online migrate failure - command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=proxbarw04' root@10.10.100.153 qm start 113 --skiplock --migratedfrom proxbarw03 --migration_type secure --stateuri unix --machine pc-i440fx-2.9' failed: exit code 255
2018-03-29 06:34:43 aborting phase 2 - cleanup resources
2018-03-29 06:34:43 migrate_cancel
2018-03-29 06:34:43 ERROR: migration finished with problems (duration 00:00:01)
TASK ERROR: migration problems


I have read several forum posts and checked that the machine is set to not remember the disks when machines are removed and restored and both host machines have both been rebooted since we started getting this error but it is still occuring.


Both host servers are running the latest version of Proxmox V5 and pull their updates down from the paid repository.

Any ideas on what the issue could be here?

Thanks,
Luca

 

[wolfgang]

Hi,

is this a shared LVM on iscsi disk?
If so is the LVM marked as shared?

 

[Luca Malaspina]

Hi,

is this a shared LVM on iscsi disk?
If so is the LVM marked as shared?

Hi,

Yes I can confirm that it is shared LVM on ISCSI and it is also marked as 'Shared' in the storage settings.

Thanks

 

[Manxmann]

Hey, sorry to drag up an old thread.

I am having the exact same problem, i.e.:

3 node cluster using external iSCSI storage (XigmaNAS)

I have migrated 8 of the 9 VPS running on node 3 to other nodes without issue however the last VPS refuses to migrate giving the following error:

2018-08-21 16:32:35 starting migration of VM 2301 to node 'px1' (10.10.0.61)
2018-08-21 16:32:35 copying disk images
2018-08-21 16:32:35 starting VM 2301 on remote node 'px1'
2018-08-21 16:32:36 can't activate LV '/dev/HAST/vm-2301-disk-1': device-mapper: create ioctl on HAST-vm--2301--disk--1LVM-PGV7kCVYyRsvonuU2FmzOC2GM4lYw80nhO85N5R2ommpc1uwwUsE1tkfk88W1S5y failed: Device or resource busy
2018-08-21 16:32:37 ERROR: online migrate failure - command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=px1' root@10.10.0.61 qm start 2301 --skiplock --migratedfrom px3 --migration_type insecure --stateuri tcp --machine pc-i440fx-2.11' failed: exit code 255
2018-08-21 16:32:37 aborting phase 2 - cleanup resources
2018-08-21 16:32:37 migrate_cancel
2018-08-21 16:32:38 ERROR: migration finished with problems (duration 00:00:03)
TASK ERROR: migration problems

I can create a new VPS using the same iSCSI external LUN and migrate this between all 3 nodes without issue. As such I don't see this as a storage issue.

All nodes are running 5.1.52 (trying to migrate in order to patch!)

I'd rather not shutdown this VPS to do an offline migration.

Any pointers much appreciated:

LV Info:

--- Logical volume ---
LV Path /dev/HAST/vm-2301-disk-1
LV Name vm-2301-disk-1
VG Name HAST
LV UUID hO85N5-R2om-mpc1-uwwU-sE1t-kfk8-8W1S5y
LV Write Access read/write
LV Creation host, time px3, 2018-08-02 14:00:09 +0100
LV Status available
# open 1
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:26

LVG Info:

--- Volume group ---
VG Name HAST
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 63
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 11
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 1024.00 GiB
PE Size 4.00 MiB
Total PE 262143
Alloc PE / Size 117248 / 458.00 GiB
Free PE / Size 144895 / 566.00 GiB
VG UUID PGV7kC-VYyR-svon-uU2F-mzOC-2GM4-lYw80n

 

[AlexLup]

Happens for me as well, on a 3 node ceph cluster.. and slow migrations every other migration 85mb/sec on a 10gbit LAN