Feature Request: Show VM Migration progress

[marsian]

Hi there,

not sure if this is the right way, but let's try

One thing that bugged me for quite some time is that when migrating VMs between nodes with local storage, no progress indicator is given. I understand that maybe forecasting an overall percentage etc. might be not as easy as expected (various network parameters involved etc.), but it would be helpful if for the migration job of a storage device of the VM at least some progress of the copying job could be shown in the task details (amount of data written), similar to the status when performing a backup?

Right now we're doing our migration forecasting based on Network load, but which is not as accurate as it should be, as you can imagine.

Thanks!

 

[LnxBil]

I understand that maybe forecasting an overall percentage etc. might be not as easy as expected (various network parameters involved etc.), but it would be helpful if for the migration job of a storage device of the VM at least some progress of the copying job could be shown in the task details (amount of data written), similar to the status when performing a backup?

On my system, there is a progress indicator:

[..]
migration status: active (transferred 104728736, remaining 180310016), total 286072832)
[..]

 

[marsian]

Interesting, we're currently still running a recent 5.4.x cluster, and there's no progress indication:

2020-03-14 23:07:38 starting migration of VM 103 to node 'pveXXX' (XX.XX.XX.XX)
2020-03-14 23:07:38 found local disk 'local-lvm:vm-103-disk-0' (in current VM config)
2020-03-14 23:07:38 copying disk images
  Using default stripesize 64.00 KiB.
  For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
  Logical volume "vm-103-disk-0" created.
  WARNING: Sum of all thin volume sizes (2.23 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (1.64 TiB)!

(Ignore the thin-pool message )

It stays like that until it's finished, and where some additional feedback will be given (taken from another migration we did earlier today):

2097152+0 records in
2097152+0 records out
137438953472 bytes (137 GB, 128 GiB) copied, 1197.12 s, 115 MB/s
66+8324367 records in
66+8324367 records out
137438953472 bytes (137 GB, 128 GiB) copied, 1200.63 s, 114 MB/s
  Logical volume "vm-105-disk-0" successfully removed
2020-03-14 22:39:17 migration finished successfully (duration 00:20:03)
TASK OK

But nothing in between on the actual copying progress...

 

[LnxBil]

Interesting, we're currently still running a recent 5.4.x cluster, and there's no progress indication:

[...]

But nothing in between on the actual copying progress...

Ah, now I get what you want. I also tried on PVE 5.4 and we have a shared storage, so the actual disk image copy stuff is not seen, because there is nothing to be done. The whole migration task log is this:

2020-03-14 23:07:05 starting migration of VM 2012 to node 'proxmox3' (10.192.0.243)
2020-03-14 23:07:05 copying disk images
2020-03-14 23:07:05 starting VM 2012 on remote node 'proxmox3'
2020-03-14 23:07:08 start remote tunnel
2020-03-14 23:07:09 ssh tunnel ver 1
2020-03-14 23:07:09 starting online/live migration on tcp:10.192.0.243:60000
2020-03-14 23:07:09 migrate_set_speed: 8589934592
2020-03-14 23:07:09 migrate_set_downtime: 0.1
2020-03-14 23:07:09 set migration_caps
2020-03-14 23:07:09 set cachesize: 33554432
2020-03-14 23:07:09 start migrate command to tcp:10.192.0.243:60000
2020-03-14 23:07:10 migration status: active (transferred 104728736, remaining 180310016), total 286072832)
2020-03-14 23:07:10 migration xbzrle cachesize: 33554432 transferred 0 pages 0 cachemiss 0 overflow 0
2020-03-14 23:07:11 migration status: active (transferred 224721217, remaining 58880000), total 286072832)
2020-03-14 23:07:11 migration xbzrle cachesize: 33554432 transferred 0 pages 0 cachemiss 0 overflow 0
2020-03-14 23:07:12 migration speed: 85.33 MB/s - downtime 48 ms
2020-03-14 23:07:12 migration status: completed
2020-03-14 23:07:16 migration finished successfully (duration 00:00:12)
TASK OK

 

[marsian]

Exactly, that's what is expected for shared storage, but with local storage there's no copy indication given, so that's my request therefore

 

[dcsapak]

mhmm we could probably use 'status=progress' for one of the dd commands

you can file an enhancement request here: https://bugzilla.proxmox.com

 

[Nothing4You]

Since I couldn't find one in bugzilla so far here we go: https://bugzilla.proxmox.com/show_bug.cgi?id=3004

 

[rnmkr]

This feature is much needed. After 1 hour I'm still wondering what's the progress of the migration.

I have found a way to show progress of the migration!

If you run pkill -USR1 dd command on the proxmox server, it reports the current migration status on the web output.

Thanks to this guy.

 

[q16marvin]

exaktly what i also search, would be quit cool!

 

[Dominic]

Note that by adding yourself to the CC list (located at top right) of the respective enhancement request on Bugzilla you will get email notifications when something interesting happens. Additionally, there is also a button to watch this forum thread above the opening post.

 

[whatthefiswrongwithcluste]

Almost two years after the feature request was opened and still no progress on such a basic functionality

 

[dmkjr]

Any movement here? For some reason my migration window stops updating after a minute or two. But I would really like to me able to see this from another screen -- terminal should I need to check the status. Right now, I'm not actually sure if it is migrating efficiently or not.

 

[dankkster]

It must be super hard to add this functionality because they have literally done everything else besides this..... #perplexed

whats worse is you dont really know if its actually doing anything or froze up. There should be progress indicators for all of the basic functions (clone, backup, migrate, etc) if its possible (and I know it is).

 

[fabian]

it's not as trivial as it sounds - a storage migration can take quite long, so it needs some sort of "backoff" logic to not blow up the task log while still giving useful information. add to that that the actual tansfer (and thus also progress information) is done by plugins (including third-party ones outside of our control) with a stable interface that we can't just break willy-nilly.

I just updated the bug tracker entry with the proposed patch implementing this for dd-based storage migration (which covers most directory-based storages and LVM/LVM-thin).

 

[LnxBil]

It must be super hard to add this functionality because they have literally done everything else besides this..... #perplexed

Yes, that is the case of ALL software. Just adding some progress bar is a no-brainer, but adding a progress bar that actually reflects an acurate progress is very hard to nearly impossible. There are so many things that can influence the progress including just reporting about it. Every progress bar will slow down the transfer and that's why there are so many memes out there about e.g. windows file transfer dialoges with years of reported time etc.