Migration of KVM doesnt work

[jompsi]

Hi all

I have a question about the migration of a kvm.

If i try to migrate a kvm i get the following error:

Apr 13 18:51:01 ERROR: migration aborted (duration 00:00:00): can't activate LV '/dev/data1/vm-102-disk-1': One or more specified logical volume(s) not found.
TASK ERROR: migration aborted

I see there is an error with my storage.

How the actual situation looks like. I have a HP DL360 and HP DL380 server. On both of them is proxmox 2.3 installed. This two servers are a two node cluster. They both have a local disk with 180GB. Further i added on both servers a second harddisk with 280GB. I added this disk like it is explained here:
http://pve.proxmox.com/wiki/Storage_Model#LVM_Groups_with_Local_Backing

These disks were added on both servers the same way with the same volume group name. The name is data1.

Is it impossible to move a kvm which disk is on data1? Is there a trick to do this?

Or should i share this disks as nfs, so the other server can access it?

pveversion -v:

pve-manager: 2.3-12 (pve-manager/2.3/ad9c5c05)
running kernel: 2.6.32-18-pve
proxmox-ve-2.6.32: 2.3-88
pve-kernel-2.6.32-18-pve: 2.6.32-88
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-17
pve-firmware: 1.0-21
libpve-common-perl: 1.0-48
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-6
ksm-control-daemon: 1.1-1

storage.cfg:

dir: local
        path /var/lib/vz
        content images,iso,vztmpl,backup,rootdir
        maxfiles 0

lvm: data1
        vgname data1
        shared
        content images

Do you need any further informations?

Thanks in advance for your support.

Have a nice day and regards
jompsi

 

[hotwired007]

are you doing an online or offline migration?

 

[tom]

dir: local
        path /var/lib/vz
        content images,iso,vztmpl,backup,rootdir
        maxfiles 0

lvm: data1
        vgname data1
        shared
        content images

you marked the lvm as shared, but it looks its not.

if you use iSCSI or FC lun, you can use this LUN for LVM and then you can mark is as shared and you can do live migrations.
see http://pve.proxmox.com/wiki/Storage_Model

 

[jompsi]

Morning

@hotwired007:
This error appears if i do an online migration. If i try a offline migration i see that the vm is moved to the other node, but i cant start it and the error says, it cant find the virtual disk of the vm. And than i cant move back the vm with the web gui. I get again the error, that the virtual disk doesnt exist on /dev/data1. Then i copied the conf File of the vm manually back to the origin node and after this i can start the vm again, cause the virtual disk was still on the first node.

@tom:
The lvm data1 is local.
Is there no other way to migrate kvm's whitout a central storage? OpenVZ can only be created on the dir: local and be migrated from there, right? KVM's can be created more or less on every disk on the server, but the migration only works if it is created on a central storage?

Thanks and regards
jompsi

 

[jompsi]

Hi all

I made some further tests but i have still some questions.

First what i found out:
- had to remove the flag "shared" in teh storage.cfg
- kvm on local can be migrated offline
- kvm on lvm(in my case "data1") cant be migrated. Doesnt matter if the server is on or off.

Error: migration aborted (duration 00:00:01): Failed to sync data - can't migrate 'data1:vm-108-disk-1' - storagy type 'lvm' not supported

In this case i have no advantage of the second harddisk. What can i do with the second harddisk, that i can do offline migrations?

Thats how the the volume configuration looks:

pvdisplay:

  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               data1
  PV Size               279.36 GiB / not usable 3.65 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              71515
  Free PE               69467
  Allocated PE          2048
  PV UUID               G1N0iS-iFJN-AKUh-1JvB-5IA1-TMD5-PBOZr8

  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               pve
  PV Size               278.87 GiB / not usable 2.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              71389
  Free PE               4095
  Allocated PE          67294
  PV UUID               lcPOHC-VV0P-yDVp-ChAo-nN9U-3HrD-aTwtAO

vgdisplay:

  --- Volume group ---
  VG Name               data1
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               279.36 GiB
  PE Size               4.00 MiB
  Total PE              71515
  Alloc PE / Size       2048 / 8.00 GiB
  Free  PE / Size       69467 / 271.36 GiB
  VG UUID               0LYN4X-bN9f-NgW5-QW5P-oAks-8uAR-678DYd

  --- Volume group ---
  VG Name               pve
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               278.86 GiB
  PE Size               4.00 MiB
  Total PE              71389
  Alloc PE / Size       67294 / 262.87 GiB
  Free  PE / Size       4095 / 16.00 GiB
  VG UUID               f74YQD-QUaf-mvVS-QUis-2kRG-LAta-HTKpQz

lvdisplay:

  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                hVzdW4-plau-x86e-LDBU-FA54-zPdh-6hrAMZ
  LV Write Access        read/write
  LV Creation host, time proxmox, 2013-03-31 12:32:25 +0200
  LV Status              available
  # open                 1
  LV Size                11.00 GiB
  Current LE             2816
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                hekQyK-fpif-2aQZ-cObU-2jgE-3Ibt-xeW2R3
  LV Write Access        read/write
  LV Creation host, time proxmox, 2013-03-31 12:32:26 +0200
  LV Status              available
  # open                 1
  LV Size                69.75 GiB
  Current LE             17856
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/pve/data
  LV Name                data
  VG Name                pve
  LV UUID                DTvTSH-PqIq-0cdP-rRzQ-yb6M-vem9-4XeT1Z
  LV Write Access        read/write
  LV Creation host, time proxmox, 2013-03-31 12:32:26 +0200
  LV Status              available
  # open                 1
  LV Size                182.12 GiB
  Current LE             46622
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

Thanks and regards
jompsi

 

[keyjey]

Hi, I have thyis same problem, bu I use local LVM and NFS filesystems, I can't migrate VMs online

 

[jompsi]

Hello

If you create a new VM and you locate the harddisk of this VM on the nfs filesystem, the online migration should work. What kind of error do you get in this scenario?

 

[keyjey]

I always get this error:

Jul 23 04:31:18 starting migration of VM 704 to node 'ca05' (xx.xx.xx.xx)
Jul 23 04:31:18 copying disk images
Jul 23 04:31:18 starting VM 704 on remote node 'ca05'
Jul 23 04:31:18 can't activate LV '/dev/pve/vm-704-disk-1': One or more specified logical volume(s) not found.
Jul 23 04:31:18 ERROR: online migrate failure - command '/usr/bin/ssh -o 'BatchMode=yes' root@xx.xx.xx.xx qm start 704 --stateuri tcp --skiplock --migratedfrom ca04 --machine pc-i440fx-1.4' failed: exit code 255
Jul 23 04:31:18 aborting phase 2 - cleanup resources
Jul 23 04:31:18 migrate_cancel
Jul 23 04:31:19 ERROR: migration finished with problems (duration 00:00:01)
TASK ERROR: migration problems

I'm sure I'm doing something wrong.

I use LVM partitions, but since VM disks are mounted in the LVM by the system, I don't know how to share it usinf NFS.

Should I just move VMs in a partition inside LVM and mount it as a NFS sistem in proxmox ?

 

[jompsi]

Thats my log if I make an online migration. But the disk of the VM is on the nfs share. In my case this nfs share is on a NAS.

Jul 23 09:56:43 starting migration of VM 108 to node 'sandy' (x.x.x.x)
Jul 23 09:56:43 copying disk images
Jul 23 09:56:43 starting VM 108 on remote node 'sandy'
Jul 23 09:56:45 starting migration tunnel
Jul 23 09:56:45 starting online/live migration on port 60000
Jul 23 09:56:45 migrate_set_speed: 8589934592
Jul 23 09:56:45 migrate_set_downtime: 0.1
Jul 23 09:56:47 migration status: active (transferred 69517949, remaining 107565056), total 2156265472)
Jul 23 09:56:49 migration status: active (transferred 212288090, remaining 0), total 2156265472)
Jul 23 09:56:50 migration speed: 409.60 MB/s - downtime 176 ms
Jul 23 09:56:50 migration status: completed
Jul 23 09:56:53 migration finished successfuly (duration 00:00:10)
TASK OK

I can see in your log, that the disk of the VM is local on a proxmox server. In this case an online migration cant work. An online migration works like this:
VM Disk is on a central Storage, like a NAS or fileserver. Your Proxmox servers have this share added.
.conf-Files are on server1. If you make an online migration the .conf-Files are moved to server2, but the harddisk of the VM stays on the nas.

I had the same problem with the LVM, so i created in the volume group a logical volume and I mounted this logical volume as a Directory on the proxmox server and on this Directory I created the VM disks. If you make it this way you can only do offline migrations.

I hope this helps. I cant guarantee that this is the right solution, but it works for me. If someone else has improvements advices for my way, I'd like to hear them

 

[keyjey]

Well, after some hours trying to understand .... I got the right formula !

I just created a Logical Volume called KVM in every server and then shared it with NFS i three nodes, added the NFS share in the cluster and voila !!!

Now it works very good, but I spent weeks to understand how this work, specially coz I'm not a linux expert, I have limited knowledge, but specially coz I didn't have experience with LVM. I was thinking that LVM could make it all and was automatically managed as a "shared storage" by the system !!!

Thanks for your help jompsi !!! you really inspired me !!

 

[arifkhumaidi]

hi all, i tried like u advice, but i got error like this

Aug 17 19:24:12 starting migration of VM 106 to node 'proxmox02' (192.168.123.14)
Aug 17 19:24:12 copying disk images
Aug 17 19:24:12 starting VM 106 on remote node 'proxmox02'
Aug 17 19:24:13 starting ssh migration tunnel
Aug 17 19:24:13 ssh tunnel still running - terminating now with SIGTERM
Aug 17 19:24:14 ERROR: online migrate failure - can't open migration tunnel - no quorum on target node
Aug 17 19:24:14 aborting phase 2 - cleanup resources
Aug 17 19:24:14 migrate_cancel
Aug 17 19:24:15 ERROR: migration finished with problems (duration 00:00:03)
TASK ERROR: migration problems

 

[udo]

no quorum on target node

Hi,
looks that your node/cluster isn't healty.

What is the output of

pvecm status

on both nodes?

Udo

 

[arifkhumaidi]

proxmox01

Version: 6.2.0Config Version: 6
Cluster Name: proxmoxcluster
Cluster Id: 12596
Cluster Member: Yes
Cluster Generation: 636
Membership state: Cluster-Member
Nodes: 2
Expected votes: 2
Total votes: 2
Node votes: 1
Quorum: 2
Active subsystems: 6
Flags:
Ports Bound: 0 177
Node name: proxmox01
Node ID: 1
Multicast addresses: 239.192.49.101
Node addresses: 192.168.123.7

proxmox02

Version: 6.2.0Config Version: 6
Cluster Name: proxmoxcluster
Cluster Id: 12596
Cluster Member: Yes
Cluster Generation: 636
Membership state: Cluster-Member
Nodes: 2
Expected votes: 2
Total votes: 2
Node votes: 1
Quorum: 2
Active subsystems: 6
Flags:
Ports Bound: 0 177
Node name: proxmox02
Node ID: 2
Multicast addresses: 239.192.49.101
Node addresses: 192.168.123.14

 

[arifkhumaidi]

wiw, after reboot, verything work fine :/, don't know why.

Aug 18 04:51:54 starting migration of VM 106 to node 'proxmox02' (192.168.123.14)
Aug 18 04:51:54 copying disk images
Aug 18 04:51:54 starting VM 106 on remote node 'proxmox02'
Aug 18 04:51:55 starting ssh migration tunnel
Aug 18 04:51:56 starting online/live migration on localhost:60000
Aug 18 04:51:56 migrate_set_speed: 8589934592
Aug 18 04:51:56 migrate_set_downtime: 0.1
Aug 18 04:51:58 migration status: active (transferred 28445458, remaining 3380183040), total 4303839232)
Aug 18 04:52:00 migration status: active (transferred 52003836, remaining 600018944), total 4303839232)
Aug 18 04:52:02 migration status: active (transferred 75752043, remaining 196980736), total 4303839232)
Aug 18 04:52:04 migration status: active (transferred 99154066, remaining 173273088), total 4303839232)