VMs mass migration, from Libvirt to Proxmox

[paradigmatic]

My organisation is about to migrate circa 100 VMs, from a bunch of libvirt machines to a brand new PVE cluster. We found a few pointers about the migration process, but it requires a lot of manual intervention.

Is there a way of automatize the process as much as possible ?

 

[LnxBil]

Is there a way of automatize the process as much as possible ?

There is no yes/no answer to that. It all that depends on the actual manual interventions and your proficiency in automating stuff.

Maybe you can describe what you tried and what manual intervention steps your need to take.

 

[paradigmatic]

The only method I found (until now) is:

1. Convert the disk image format from RAW to QCOW
2. Copy the image disk to the PVE node storage
3. Create a VM in PVE with similar config
4. Import the disk image with qm disk import...
5. Change the VM config to use the new disk image
6. Discard the old disk image

I found this process in: https://malacube.wordpress.com/2022/02/21/kvm-qemu-migrate-to-proxmox/

I would like to avoid repeating those steps for the 100 VMs...

Our proficiency in automating stuff: we have software engineering skills, so we are comfortable with scripting or using a remote API. However, I'd rather avoid spending time writing scripts to automatize a sub-par solution if a better one exists.

 

[Johannes S]

The official documentation is this one:
https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE

Depending on your needs there are several options, the easiest is propably using the import wizard:
https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Automatic_Import_of_Full_VM

The manual methods are more envolved but (depending which you choose) can be faster or have a smaller downtime e.G. this one:

https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Attach_Disk_&_Move_Disk_(minimal_downtime)

 

[paradigmatic]

The official documentation is this one:
https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE

Depending on your needs there are several options, the easiest is propably using the import wizard:
https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Automatic_Import_of_Full_VM

The manual methods are more envolved but (depending which you choose) can be faster or have a smaller downtime e.G. this one:

https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Attach_Disk_&_Move_Disk_(minimal_downtime)

Thanks for your answer. We read the official doc. Most of the procedure you linked are targeted at ESXi systems, however we are migrating from libvirt settings.

I should also add that some downtime is not issue.

 

[bbgeek17]

1. Convert the disk image format from RAW to QCOW
2. Copy the image disk to the PVE node storage
3. Create a VM in PVE with similar config
4. Import the disk image with qm disk import...
5. Change the VM config to use the new disk image
6. Discard the old disk image

Which part of it do you want to avoid?
You don't have to convert to qcow, you can use raw. You'd loose some functionality, so is that something you want to save?
You have to get the disk to PVE, so you probably can't avoid it. Are disks on NFS now and staying on NFS, you can move in place.
You have to create a VM, cant avoid it.
You don't need to import. You can just place the file into the right location with the right name. Of course, that depends on type of storage used.
Running a command to point to new disk doesn't seem like it is worth optimizing.
If you don't do import, you don't need to discard. At the same time it's just an "rm".

So the question is : what are you trying to optimize in these 6 steps?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[paradigmatic]

I am not trying to optimize each step in itself, but rather trying to understand:

• Is there a alernative process with fewer steps ?
• OR ELSE, can I automatize the whole process ?

For instance, I could automatize some of the steps, by parsing the libvirt domain XML file to extract the image location and the VM configuration. With these data I could script the rest of the process. I did not find such script. I am willing to write it (and then share it), but I would like to know if some ready made solution exist, or any other alternate procedure before starting.

Although it may seem pointless, we'll have to run the whole process 100 times. Such a high number of boring repeated task is susceptible to error.

 

[news]

can you run the client of proxmox backup server on your old system to save it?
then restore it on your new setup and change the hardware settings.

# https://pbs.proxmox.com/docs/backup-client.html
# https://pbs.proxmox.com/docs/installation.html#install-statically-linked-proxmox-backup-client
# https://www.proxmox.com/en/downloads

 

[Johannes S]

• Is there a alernative process with fewer steps ?
• OR ELSE, can I automatize the whole process ?

In theory it should be possible to automate the minimal- downtime process with the shared storage for esxi and NFS like this:
- Use Vmware API to save the configuration of VM and move the VM Discs to the NFS share (afik this is possible without shutting down the VM)
- Use ProxmoxVE API or CLI tools to create new vms with the saved values from VMware, put it's disc images on the same NFS storage as ESXI
- Shutdown the VM (again vmware api) in ESXI, copy the vmdk files and edit them as described in https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Attach_Disk_&_Move_Disk_(minimal_downtime)
- Power on the VM in Proxmox, afterwards move the VM discs to the target storage (you don't need to shutdown it)

In the German forum @Falk R. presented a batch file together with a VIRTIO driver method to install the needed virtio drivers on WindowsVMs before the migration which is quite handy for minimal downtimes and can also be used for automation (by calling the batchfile before starting the rest of the procedure). He also mentioned that some of his customers automated everything but not how: https://forum.proxmox.com/threads/kleiner-gamechanger-bei-der-windows-migration-zu-proxmox.167837/

Maybe he can chime in, how to achieve this?

Please note that I have no practical experience, but from everything I read it should be possible to create a script/ansible play or whatever automation tool you use to automate the needed steps.
Edit: An alternative to the batch file + custom iso method is using dism like described here: https://forum.proxmox.com/threads/m...te-windows-vms-from-vmware-to-proxmox.163855/

 

[bbgeek17]

Use Vmware

Op is using Linux LibVirt (Redhat?) hypervisor. There is no VMware API.

@paradigmatic, frankly, it’s unlikely that a tool or process exists for what you’re looking for if you haven’t already found one. There was a large wave of people (and money) moving from ESXi to PVE, so the market responded - Proxmox built an automated migration tool for that.

By contrast, there have been far fewer migrations from lateral hypervisors like LibVirt. These did not have the same visibility and once people complete such transitions, they tend to move on to automating longer-term projects instead.

If you happen to find or create something yourself, please do update the thread.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Johannes S]

Op is using Linux LibVirt (Redhat?) hypervisor. There is no VMware API.

Sorry my bad. You are of course right, didn't know how I could mess this up.
But imho the basic method should work the same way: Read VM config from LibVirt and create new config with PVE. Shutdown the old VM and copy the image to the location of the image configured in PVE. This should be possible manual or via a script. Ideally the old vm data is on a NFS share , because then it would be enough to edit the PVE VM config to point to the disc image on the NFS share.

 

[bbgeek17]

Sorry my bad. You are of course right, didn't know how I could mess this up.

Dangerously low amount of coffee in your bloodstream Happens to me all the time

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

Is there a alernative process with fewer steps ?

Actually, those steps you mention are the required ones and those are also the simple ones. I thought you have problems booting the VMs due to the hardware changes and strange guest operating systems like the ones from microsoft. You are lucky however that you're already running on QEmu. With others you would need to install drivers before migrating, regenerating initrds only to mention two common ones from non-qemu-hypervisors.

can I automatize the whole process ?

Sure, almost everything is automatable. As you already said yourself, write something that can convert the xml to settings PVE can understand, the rest is easy API stuff that e.g. ansible or even plain python can do. The disk in libvirt is already in a format the PVE understands, again both use QEmu, so you don't need to convert them. As @bbgeek17 already said, if the disks are on a disk that can be shared via NFS, you don't need to do anything on that front before starting the VM. After starting up, you can live migrate the storage to the one you want to use in PVE and have a very, very brief downtime of minutes for each machine. I would however test this with copying it in order to not have to test with production data.

 

[paradigmatic]

Many thanks for all the answer. I'll write some crude scripts and make them available here in case someone is interested.