Live migration with discard

[FormerVMW]

It is discussed in the following thread, but the thread quickly devolved into a shared storage discussion: https://forum.proxmox.com/threads/live-migration-zfs-discard.143824/

I'd like to keep this thread focused solely on the topic of the thread.

When doing a live migration, with discard enabled, VMs being migrated appear to "pre-allocate" the space on the destination host/storage. After the space is pre-allocated, then the actual migration starts. This only occurs, that I can see, with discard enabled.

This is a problem because when it does the pre-allocation it consumes 100% of the disk IO on the destination machine. Which is obviously a big problem that is compounded with huge disks. For example, if a VM has 2TB of disks configured, the migration will pound the storage at the maximum possible rate until the pre-allocation of 2TB is completed.

A few questions:

1. Why does it do this with discard enabled disks?
2. Is there any way to minimize the load? i.e. set "pre-allocation" rate to some reasonable MBps and not flood the IO by default?
3. Is there any way to disable this pre-allocation without disabling discard?

 

[Londeril]

Hi

I took the advice given in the original thread and am using the "replicate then migrate" solution.

1) set a replication job from one host to another. This will be very quick and only used blocks will be transferred (the only drawback: you have to use ZFS storages on both hosts and they must be given the same name - but this is best practice in a Proxmox cluster anyways)
2) wait for the replication to finish
3) initiate a migration - this will only take a couple of minutes (or seconds).
4) remove the replication job (or as I have done: simply keep the replication job and set it to replicate twice a day - this way you'll always have a (semi) up to date version of the VM on another host.

This is as good (if not better) then live migration - well it IS live migration; as the replication and the migration after it will not take the VM offline (or just a couple of milliseconds)

I'm not sure why live-migration without a prior replication does not work when discard is enabled - but this workaround (if it even is one) went into production use in our Proxmox data center(s) and we haven't looked back since.

Cheers

 

[FormerVMW]

Hi

I took the advice given in the original thread and am using the "replicate then migrate" solution.

1) set a replication job from one host to another. This will be very quick and only used blocks will be transferred (the only drawback: you have to use ZFS storages on both hosts and they must be given the same name - but this is best practice in a Proxmox cluster anyways)
2) wait for the replication to finish
3) initiate a migration - this will only take a couple of minutes (or seconds).
4) remove the replication job (or as I have done: simply keep the replication job and set it to replicate twice a day - this way you'll always have a (semi) up to date version of the VM on another host.

This is as good (if not better) then live migration - well it IS live migration; as the replication and the migration after it will not take the VM offline (or just a couple of milliseconds)

I'm not sure why live-migration without a prior replication does not work when discard is enabled - but this workaround (if it even is one) went into production use in our Proxmox data center(s) and we haven't looked back since.

Cheers

Thanks for the post. That's an extreme "solution" and I do not use ZFS. In my situation it would be easier to restore from a PBS backup on the huge VMs. Which isn't really a solution because there is downtime involved and there should be one method for live migrations (documentation/runbooks, consistency, etc etc) . The replication method is framed as a solution in the manual, but for our use it's a stop-gap/kludge to compensate for a defect with how discard enable disks are migrated. https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvesr

Live migrations should be universal without having to document caveats. e.g. "Please note that if discard is enabled, you'll crush the storage on the destination host while the disk is being created". I'll await a response from the Proxmox guys to see if there is a realistic option for those not using ZFS. At a minimum there should be mention of this situation when marking disks with discard. No mention here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_hard_disk_discard nor here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_migration

 

[LnxBil]

This is as good (if not better) then live migration

Care to elaborate? Having a shared storage is still superior to ZFS replication. It just works instantly with respect to disk. No need to copy at all.

 

[LnxBil]

Live migrations should be universal without having to document caveats. e.g. "Please note that if discard is enabled, you'll crush the storage on the destination host while the disk is being created". I'll await a response from the Proxmox guys to see if there is a realistic option for those not using ZFS. At a minimum there should be mention of this situation when marking disks with discard. No mention here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_hard_disk_discard nor here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_migration

If you want proper live migration (and also failover), just use a shared storage. That is the enterprise way to go, gas been and always will be. ZFS replication is nice and all, but not on the same level. Share-Nothing-Migration is maybe acceptable for some corner case, but not in production environment ...

 

[FormerVMW]

If you want proper live migration (and also failover), just use a shared storage. That is the enterprise way to go, gas been and always will be. ZFS replication is nice and all, but not on the same level. Share-Nothing-Migration is maybe acceptable for some corner case, but not in production environment ...

Read the first line and second line of my original post.