[TUTORIAL] online vm migration from remote KVM with block device backend to Proxmox ( any kind of supported storage )

[tm77nrj]

Hello all,


Lately I was searching for a solution to live / online migrate KVM VMs with block device backend (DISK / LVM / SAN ) to Proxmox ( any kind of storage that is supported / configured ).
Because there are users that have huge VMs in production and downtime is not an option ( so dd is not a workable scenario ), I tried to put together a solution so I can migrate also this clients to Proxmox.

Of course, this approach depends on your current setup, but with this information I think that you can adapt to your environment.

Without further introduction, the steps are this:
0. Backup
1. On source physical server where you have KVM, install nbd-server or nbdkit.
2. Shutdown the VM.
3. Export the VM backing device (or devices) with something like: nbdkit -p 9999 file /dev/vgname/lvname
You can export multiple disks, but with different ports. You can export /dev/mapper/sandisk or local raw disk or any raw block device.
Secure transfer between the hosts or any other security measures is in your hand. My recommendation is at least setup some iptables rules on the source host so there will be no traffic with those ports from any other host except the destination Proxmox.
4. On target Proxmox, install nbd-client or nbdkit. It's up to you which version of NBD better suites your need.
apt install nbd-client
5. Map the remote disk locally:
nbd-client remoteip 9999 /dev/nbd0
6. Create a temporary storage if your current Proxmox does not support file based VM disk images.
Something like: create /tmpfs folder , Datacenter -> Storage -> Add Directory -> /tmpfs ...
7. Create the VM that you want to migrate, but without backing disk. I recommend to clone also the MAC address from the source VM.
Get the VM ID.
8. In Proxmox CLI:
Create a folder like /tmpfs/images/vmid
in this folder:
ln -s /dev/nbd0 transfer0.raw
chmod 666 /dev/nbd0 (adapt to your specific needs)
qm rescan
9. In Proxmox GUI, go to your VM, double click on the unused disk and set it up accordingly, In hardware change the boot order to boot from this disk.
10. I didn't mention anything about drivers or other problems with the migration, because this are self explanatory that they are needed.
11. Boot the VM
12. Disk actions -> Move storage
13. TEST
14. Clean up: remove unusued disk, Remove Storage tmpfs, nbd-client -d /dev/nbd0
15. Enjoy and give a reply if you noticed a problem.

 

[einhirn]

I like this one, and I expanded on it to use a similar method for VM migration from VMware (or any hypervisor that allows adding existing disks to a helper VM).

On the linux helper vm, I'm using "nbdkit -p 9999 --directory=/dev" to just export every block device that helper vm has by name.
I can then map the remote disk locally by giving it's name (i.e. sdc and so on) to nbd-client.

That way you don't need multiple ports and also don't need to touch the helper VM each time you change the disks you want to migrate.

When using the PVE api to create the new VM, I can even pre-add the softlinks for nbdX into the VM's directory beforehand and simply refer to those in the create call.

I'll "move storage" via GUI, using a convoluted one-liner that will migrate every disk which still is on the source storage one after another, since we can only move one disk at a time...

export ID=111; qm config $ID | grep scsi[0-9]*:.*local:$ID | cut -d: -f1 | xargs -iREPLACE -n1 -t qm disk move $ID REPLACE local_zfs01 --delete

In this example, the source storage is 'local' (a "directory"-type file storage) while the target is 'local_zfs01' but can be replaced by probably any PVE storage name.

I'm also working on a script to manage the manual steps as automatic as possible - but I still need to update source VM's network config and as a safety measure run 'blkid /dev/sd*' there anyway. I'll check the mapped NBDs on the PVE host with blkid, too, before firing up the newly created VM.