Live migration proxmox KVM without shared storage

[ash]

Hi,

I would like to live migrate my proxmox KVM VM to another node without shared storage.The storage I am using is LVM based storage. If i try to do migrate using LVM based storage having same storage name within the cluster with qm migrate 1009 prox2 --online --with-local-disks, the migration does start but ends with error, the VM though is migrated to the new node.


2020-06-05 13:50:15 migration speed: 3.05 MB/s - downtime 15 ms
2020-06-05 13:50:15 migration status: completed
2020-06-05 13:50:20 ERROR: removing local copy of 'vg2:vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if' failed - lvremove 'vg_prox1/vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if' error: Logical volume vg_prox1/vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if in use.
drive-sata0: transferred: 2147549184 bytes remaining: 0 bytes total: 2147549184 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
drive-sata0: Completing block job...
drive-sata0: Completed successfully.
drive-sata0 : finished
2020-06-05 13:50:21 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=prox2' root@ipaddr pvesr set-state 1009 \''{}'\'
Logical volume "vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if" successfully removed
2020-06-05 13:50:26 ERROR: migration finished with problems (duration 00:03:31)
migration problems

And if i try to migrate the LVM based VM without shared storage and having different storage names on both nodes then i encounter this error

2020-06-05 13:45:54 copying local disk images
2020-06-05 13:45:54 starting VM 1008 on remote node 'prox1'
2020-06-05 13:45:55 [prox1] no such volume group 'v2'
2020-06-05 13:45:55 ERROR: online migrate failure - remote command failed with exit code 255
2020-06-05 13:45:55 aborting phase 2 - cleanup resources
2020-06-05 13:45:55 migrate_cancel
2020-06-05 13:45:56 ERROR: migration finished with problems (duration 00:00:03)
migration problems


Any suggestion how can i do this?

 

[fabian]

please provide more details:
- pveversion -v output
- storage.cfg
- VM config
- full migration task logs

 

[ash]

My prox1 node has : local storage, lvm raw storage
Prox2 node has: local strage, lvm raw storage , lvm scsi storage

both nodes having same pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-helper: 6.1-6
pve-kernel-5.3: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-13
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-21
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-3
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-6
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,vztmpl,backup

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

dir: vms_file
path /vms
content rootdir,images

lvm: vg1
vgname vg_prox1
content rootdir,images

lvm: vg2
vgname vg_prox1
content images,rootdir

lvm: v2
vgname v2
content images,rootdir

 

[ash]

Migration:-
VM 1009 is created on prox 1 using LVM RAW storage type, prox2 alos has this type of storage but none are shared.

root@prox1:~# qm migrate 1009 prox2 --online --with-local-disks
2020-06-05 13:46:55 starting migration of VM 1009 to node 'prox2' (10.0.0.13)
2020-06-05 13:46:56 found local disk 'vg1:vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if' (in current VM config)
2020-06-05 13:46:56 found local disk 'vg2:vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if' (via storage)
2020-06-05 13:46:56 copying local disk images
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 145.993 s, 14.7 MB/s
20846+29479 records in
20846+29479 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 145.255 s, 14.8 MB/s
2020-06-05 13:49:25 starting VM 1009 on remote node 'prox2'
2020-06-05 13:49:32 start remote tunnel
2020-06-05 13:49:33 ssh tunnel ver 1
2020-06-05 13:49:33 starting storage migration
2020-06-05 13:49:33 sata0: start migration to nbd:10.0.0.13:60001:exportname=drive-sata0
drive mirror is starting for drive-sata0
drive-sata0: transferred: 0 bytes remaining: 2147483648 bytes total: 2147483648 bytes progression: 0.00 % busy: 1 ready: 0
drive-sata0: transferred: 91226112 bytes remaining: 2056257536 bytes total: 2147483648 bytes progression: 4.25 % busy: 1 ready: 0
drive-sata0: transferred: 229638144 bytes remaining: 1917845504 bytes total: 2147483648 bytes progression: 10.6 9 % busy: 1 ready: 0
drive-sata0: transferred: 281018368 bytes remaining: 1866465280 bytes total: 2147483648 bytes progression: 13.0 9 % busy: 1 ready: 0
drive-sata0: transferred: 332398592 bytes remaining: 1815085056 bytes total: 2147483648 bytes progression: 15.4 8 % busy: 1 ready: 0
drive-sata0: transferred: 400556032 bytes remaining: 1746927616 bytes total: 2147483648 bytes progression: 18.6 5 % busy: 1 ready: 0
drive-sata0: transferred: 458227712 bytes remaining: 1689255936 bytes total: 2147483648 bytes progression: 21.3 4 % busy: 1 ready: 0
drive-sata0: transferred: 693108736 bytes remaining: 1454374912 bytes total: 2147483648 bytes progression: 32.2 8 % busy: 1 ready: 0
drive-sata0: transferred: 912261120 bytes remaining: 1235222528 bytes total: 2147483648 bytes progression: 42.4 8 % busy: 1 ready: 0
emaining: 1085341696 bytes total: 2147549184 bytes progression: 49. 46 % busy: 1 ready: 0
drive-sata0: transferred: 1136656384 bytes remaining: 1010892800 bytes total: 2147549184 bytes progression: 52. 93 % busy: 1 ready: 0
drive-sata0: transferred: 1443889152 bytes remaining: 703660032 bytes total: 2147549184 bytes progression: 67.2 3 % busy: 1 ready: 0
drive-sata0: transferred: 1928331264 bytes remaining: 219217920 bytes total: 2147549184 bytes progression: 89.7 9 % busy: 1 ready: 0
drive-sata0: transferred: 2114977792 bytes remaining: 32571392 bytes total: 2147549184 bytes progression: 98.48 % busy: 1 ready: 0
drive-sata0: transferred: 2147549184 bytes remaining: 0 bytes total: 2147549184 bytes progression: 100.00 % bus y: 0 ready: 1
all mirroring jobs are ready
2020-06-05 13:50:14 starting online/live migration on tcp:10.0.0.13:60000
2020-06-05 13:50:14 set migration_caps
2020-06-05 13:50:14 migration speed limit: 8589934592 B/s
2020-06-05 13:50:14 migration downtime limit: 100 ms
2020-06-05 13:50:14 migration cachesize: 16777216 B
2020-06-05 13:50:14 set migration parameters
2020-06-05 13:50:14 start migrate command to tcp:10.0.0.13:60000
2020-06-05 13:50:15 migration speed: 3.05 MB/s - downtime 15 ms
2020-06-05 13:50:15 migration status: completed
2020-06-05 13:50:20 ERROR: removing local copy of 'vg2:vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if' failed - lvre move 'vg_prox1/vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if' error: Logical volume vg_prox1/vm-1009-dpnqeng8stjw vi3x-zsornioffigt90if in use.
drive-sata0: transferred: 2147549184 bytes remaining: 0 bytes total: 2147549184 bytes progression: 100.00 % bus y: 0 ready: 1
all mirroring jobs are ready
drive-sata0: Completing block job...
drive-sata0: Completed successfully.
drive-sata0 : finished
2020-06-05 13:50:21 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=prox2' root@10.0.0.13 pvesr set- state 1009 \''{}'\'
Logical volume "vm-1009-dpnqeng8stjwvi3x-zsornioffigt90if" successfully removed
2020-06-05 13:50:26 ERROR: migration finished with problems (duration 00:03:31)
migration problems

VM 1008 is created on prox2 with LVM SCSI storage type. prox 1 does not have this type of storage.
root@prox2:~# qm migrate 1008 prox1 --online --with-local-disk --force
2020-06-05 13:45:53 starting migration of VM 1008 to node 'prox1' (10.0.0.12)
2020-06-05 13:45:54 found local disk 'scsi-v2:vm-1008-dlnu03xgezemwfrb-kgtk75svvxndvjx0' (in current VM config)
2020-06-05 13:45:54 copying local disk images
2020-06-05 13:45:54 starting VM 1008 on remote node 'prox1'
2020-06-05 13:45:55 [prox1] no such volume group 'scsi_v2'
2020-06-05 13:45:55 ERROR: online migrate failure - remote command failed with exit code 255
2020-06-05 13:45:55 aborting phase 2 - cleanup resources
2020-06-05 13:45:55 migrate_cancel
2020-06-05 13:45:56 ERROR: migration finished with problems (duration 00:00:03)
migration problems

 

[fabian]

you have two storage entries (vg1 and vg2) referring to the same volume group. this is not supported and leads to all sorts of issues - it also does not make much sense. for the second case you'd need to provide a targetstorage to switch storage during migration. if a local storage is only available on some nodes in the cluster, you should explicitly set this in storage.cfg (with 'nodes').

 

[ash]

you have two storage entries (vg1 and vg2) referring to the same volume group. this is not supported and leads to all sorts of issues - it also does not make much sense. for the second case you'd need to provide a targetstorage to switch storage during migration. if a local storage is only available on some nodes in the cluster, you should explicitly set this in storage.cfg (with 'nodes').

vg1 is the storage available with prox1 and vg2 is with prox2. The storage name is same. Does this cause issue?

 

[fabian]

vg1 is the storage available with prox1 and vg2 is with prox2. The storage name is same. Does this cause issue?

yes. you tell PVE "I have a storage called vg1 using VG vg_prox1 on all nodes and another storage called vg2 using VG vg_prox1 on all nodes", and PVE thinks that the volume 'foo' on vg1 is a different volume than 'foo' on vg2. what you want is a single "storage 'foobar' using VG vg_prox1 on all nodes", but you might need to adapt existing guest configs to point to the right place.

 

[ash]

yes. you tell PVE "I have a storage called vg1 using VG vg_prox1 on all nodes and another storage called vg2 using VG vg_prox1 on all nodes", and PVE thinks that the volume 'foo' on vg1 is a different volume than 'foo' on vg2. what you want is a single "storage 'foobar' using VG vg_prox1 on all nodes", but you might need to adapt existing guest configs to point to the right place.

thanks for the guidance. I had another issue.

I have a vm in prox1 and i wish to migrate it to a new vm in prox2. For eg prox1 has 1001 the vm to migrate. I create a dummy vm1002 with the same configuration as 1001 in prox2 now i want to migrate 1001 to 1002. Is that possible? If it is, then how can I do that using command.

 

[fabian]

you need to use backup + restore to create a new VM with the same content as another one, but a different ID.