Moving non-shared storage VM to another node

e100

Renowned Member
Nov 6, 2010
1,268
45
88
Columbus, Ohio
ulbuilder.wordpress.com
Today I needed to move a VM from the local storage on one node, to the local storage on another node with little downtime.
Decided to test a utility called lvmsync. https://github.com/mpalmer/lvmsync

The VM I was moving had two virtual disks stored on lvm volumes.
I first created an identical VM on another node.

Source VM is on nodeA id 130, Destination is on nodeB id 198

After installing lvmsync on both nodes this is what I did:

Created a snapshot on the source for the two VM disks:
Code:
root@nodeA:/ # lvcreate --snapshot -L10G -n vm-130-disk-1-snap /dev/lvmstorage/vm-130-disk-1
root@nodeA:/ # lvcreate --snapshot -L100G -n vm-130-disk-2-snap /dev/lvmstorage/vm-130-disk-2

Next I used dd over SSH to clone the snapshots to the new vm destination.
Code:
root@nodeA:/ # dd if=/dev/lvmstorage/vm-130-disk-1-snap bs=1M | pv -ptrb | ssh root@nodeB dd of=/dev/lvmstorage/vm-198-disk-1 bs=1M
root@nodeA:/ # dd if=/dev/lvmstorage/vm-130-disk-2-snap bs=1M | pv -ptrb | ssh root@nodeB dd of=/dev/lvmstorage/vm-198-disk-2 bs=1M

Now I shutdown the VM, once the source VM is stopped I use lvmsync to copy the changes that were made during the initial copy.
Code:
root@nodeA:/ # lvmsync /dev/lvmstorage/vm-130-disk-1-snap nodeB:/dev/lvmstorage/vm-198-disk-1
root@nodeA:/ # lvmsync /dev/lvmstorage/vm-130-disk-2-snap nodeB:/dev/lvmstorage/vm-198-disk-2
Those two commands took about 1 minute each.

At this point you can turn the destination VM on since your data is fully cloned.
Since this was the first time ever trying this I decided to get an sha sum of one disk on the source and destination to ensure they are indeed identical.
Code:
root@nodeA:/# sha256sum -b /dev/lvmstorage/vm-130-disk-1
[B]cc5538969acb51bd73759b4c83021cdb35dab37825379659e1b5a760eb741412[/B] */dev/lvmstorage/vm-130-disk-1

root@nodeB:/# sha256sum -b /dev/lvmstorage/vm-198-disk-1
[B]cc5538969acb51bd73759b4c83021cdb35dab37825379659e1b5a760eb741412[/B] */dev/lvmstorage/vm-198-disk-1

Then I started the VM on the destination node, works perfect.


lvmsync also has some other neat features such as being able to store the changes to a file.
I am thinking that maybe with a little scripting it would be easy to setup a process that works like transaction log shipping for databases but for the whole VM.
Imagine tracking all the changes to your VM disks and every 15 minutes shipping the changes to some remote location and auto-applying those changes.

lvmsync is pretty awesome and it works well.
Maybe it will help you solve an issue some day.
 
Last edited:
Hi e100,
nice tool!

Why you use the snapshot for the initial sync and not the changed partition? and how can the checksum be the same?
After the first copy the removing of the snapshot would be the right, or?

I assume nodeB is vm6?!

Udo
 
I fixed the vm6, forgot to change it to nodeB.

You can delete the snapshot after running the lvmsync command.

Disk1 was 40GB disk2 was 550GB.
I had about 10 minutes of total downtime moving this VM, most of that 10 minutes was running the checksum.
If I had skipped that I would have had only about two minutes downtime.

lvmsync looks at the metadata in the snapshot to see what blocks changed since the snapshot was made.
It then reads those blocks from the source (not the snapshot) and copies them to the destination.
This is why the initial copy is made from the snapshot/
 
Hi e100,

I am trying to do the same thing that you did, but i do not have the directory /dev/lvmstorage/ on my proxmox install! How can i tell lvm to create a snapshot of the vm disk? On my install i only see the /dev/pve directory and inside of it i can only find 3 files, data, root and swap. Thank you very much for any help.


 
The bad thing of this method for having hot-backups is that if something goes wrong on the source storage during the copy, both copies are lost!
Fixing it requires double space on the destination:
- take a snapshot of the destination LVs (on nodeB) -- size must be at least the same of the LV being copied
- proceed with copy (this will be slower, since nodeB will have to do a read-write-write instead of a simple write... cost of security!)
- do the final lvmsync
- if all went well, remove snapshot on nodeB, else you can dd back from the snapshot to the source volume (I don't remember if there's a way to say LVM to discard "master" volume and rollback to the snapshot...).

PS: taking snapshots from 2 disks while the machine is running could lead to problems. It's rare, but possible... Think if the second contains a log of transactions on the first:
- you take a snapshot of the first,
- a log gets committed: this modifies both disks
- you take a snapshot of the second disk, that no longer contains transaction data
But your image of the first disk requires that transaction to be consistent...
So it's really better to at least do a suspend/resume around taking snapshots of multiple disks: adds about 1sec but avoids these problems!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!