Live migration with local directories

Josh Douglas

Member
Dec 29, 2015
15
1
23
Please forgive the dumb question(s).

Hypothetical setup:

3 servers, set to cluster, and in quorum.
Each have their own LVM volumes where volumes are mounted with EXT4 filesystems so directories and disks images are readable/viewable as files on the file system, i.e. each server has a mounted filesystem call "/vms" but may have different sized volumes but enough space for a set of VMs (more that 500GB), and that directory has been marked as "shared" in the web interface and each server has "/vms" on each system.

For instance, VM 101's disk image exists on all systems along with the typical directories of dump, images, private and template, yet is running on 1 node at a time.

If all disk images reside on each of the servers in the quorum, what issues might arise with live migrations?
Do the disk images sync up while migrating?
Are there any guest OS considerations to take into account?

I realize it's probably not "efficient", looking more for reliability with limited resources. I have searched but haven't found anything like this answered. Most things I find are for setting up DBRD, NFS and have some centralized storage, which I can't say will be an option.

That being said. I have a virtual cluster setup of this type for testing before going live with a hardware setup and it "seems" to work. I'm looking for some advice into how to make this work, or if it's a fools errand. Apologies if it comes off as hand holding.

Thanks for taking the time to read this, and apologies if this is not something, "obvious" that I'm not getting.
 
it's available both on PVE 4.4 and PVE 5 beta, but only on the CLI for now (with "qm migrate VMID TARGETNODE --online --with-local-disks")

Awesome. Does it still have to be part of a cluster, or can it be migrated to another stand alone server?
 
Awesome. Does it still have to be part of a cluster, or can it be migrated to another stand alone server?
Hi,
with pve4.4 the nodes must be part of an cluster (but afaik spirit work on an version to migrate VMs to another cluster…).

One remark to your first post - the storage /vms must not defined as shared - if the dir is shared, the disk will not migrated.

Udo
 
Hi,
with pve4.4 the nodes must be part of an cluster (but afaik spirit work on an version to migrate VMs to another cluster…).

One remark to your first post - the storage /vms must not defined as shared - if the dir is shared, the disk will not migrated.

Udo

I had send prelimary patches for cross cluster migration, but it not really clean for now, (but it's works). I need to rework them to include them in proxmox, but I think it'll not be ready before proxmox 5.1.
 
I had send prelimary patches for cross cluster migration, but it not really clean for now, (but it's works). I need to rework them to include them in proxmox, but I think it'll not be ready before proxmox 5.1.

How is this coming along? Will there be an option to just do server to server without clustering?
 
I just tried the live migration via cli with a single vm disk on LVM.

TLDR: It works :)

After initially failing it turned out my firewall should have tcp 60000 open.

scsi0: start migration to to nbd:192.168.1.2:60000:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
socat[13125] E read(5, 0x5616e946e000, 8192): Connection refused

Maybe that's something to add to the 'Ports used by Proxmox VE' section on https://pve.proxmox.com/wiki/Firewall ?

Some other remarks:
  • the fact that I needed to open port 60000 tells me that the disk transfer does not benefit from SSH's encryption
  • the disk transfer went at near Gbit wirespeed, but the memory transfer capped at 10.02 MB/s. I assume that's the difference between plain vs encrypted.
  • It reported 'downtime 68 ms' but an open ping missed 6 replies(seconds). This might be the network needing to learn the new location of this VM's MAC address. Still good enough for most operations though :)
  • The scheduled backup failed on the old host after this with 'unable to find VM'

+1 for integrating this in the webUI
 
  • Like
Reactions: alamadrid
It would really be nice to be able to do this without a cluster, i.e. server to server migration. The '-migrateexternal' doesn't appear to be an option anymore, at least in 5.2.x.
 
  • Like
Reactions: alamadrid
It would really be nice to be able to do this without a cluster, i.e. server to server migration. The '-migrateexternal' doesn't appear to be an option anymore, at least in 5.2.x.

Hi,
I'm still working on it again recently, It should be officially include really soon.
 
  • Like
Reactions: alamadrid
Any updates on this?

[EDIT]
I've currently been playing around with migrations on a 6.x cluster of 3. The migration option is fast enough for my purposes, as I know in my research on this yielded some people saying that it would be "too slow", bonded NICs helps, and 3-5 minutes for my 32-64GB VMs is great for me. The few 'big ones' are in the multiple disk range with a VM of two 500GB disk images taking the longest, which I'm still measuring, but still more than acceptable for my use case.

Thanks for the great work!
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!