Slow livemigration performance since 5.0

No news about this,
I have not found any time yet.
 
Can confirm problem. The bug appeared in 5.0 with introduction of storage replication feature (and pvesr). I gueess it was update of pve-manager from 5.0-10 to 5.0-23. In PVE 5.0 beta2 live migrathion was fast as in PVE 4 with maximum 1 ping packet lost.
Problem reprodused on shared storage too, but delay is shorter.
With local storage:
Code:
2017-07-19 09:29:10 migration status: active (transferred 332452457, remaining 3585961984), total 4312604672)
2017-07-19 09:29:10 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-19 09:29:12 migration status: active (transferred 671704944, remaining 10498048), total 4312604672)
2017-07-19 09:29:12 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 128 overflow 0
2017-07-19 09:29:13 migration speed: 341.33 MB/s - downtime 15 ms
2017-07-19 09:29:13 migration status: completed
drive-scsi0: transferred: 42949672960 bytes remaining: 0 bytes total: 42949672960 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
drive-scsi0: Completing block job...
drive-scsi0: Completed successfully.
drive-scsi0 : finished
2017-07-19 09:29:25 # /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=somehost' root@1.1.1.1 pvesr set-state 9006 \''{}'\'
Ping statistics from host connected to same switch:
Code:
39 packets transmitted, 24 received, 38% packet loss, time 38891ms
15 ping packets lost

With shared Ceph:
Code:
2017-07-19 09:42:57 migration status: active (transferred 409286050, remaining 3496914944), total 4312604672)
2017-07-19 09:42:57 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-19 09:42:59 migration speed: 1024.00 MB/s - downtime 16 ms
2017-07-19 09:42:59 migration status: completed
2017-07-19 09:43:00 # /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=somehost' root@1.1.1.1 pvesr set-state 9006 \''{}'\'
2017-07-19 09:43:03 migration finished successfully (duration 00:00:11)
Ping statistics from host connected to same switch:
Code:
42 packets transmitted, 38 received, 9% packet loss, time 41002ms
4 ping packets lost.

Is it possible to simply disable this feature with pvesr if i do not plan to use ZFS and its replication?
 
I think you mix some things here.
The bug it a problem in --with-local-storage migration and has nothing to do with pvesr.
We run in a timeout what wait for 10 sec for each drive.

You ceph migration do not use this code path and your 4 package lost is may by a related the new route of the network.
This can happened because the ping is send to the old node.
 
I think you mix some things here.
The bug it a problem in --with-local-storage migration and has nothing to do with pvesr.
We run in a timeout what wait for 10 sec for each drive.

You ceph migration do not use this code path and your 4 package lost is may by a related the new route of the network.
This can happened because the ping is send to the old node.
Yes it is possible. This is why i mentioned that pinging from the same switch (L2 and no routes and this is really good fast 10G switch and little network traffic and no congestion), and that in PVE 5.0 beta2 were no delays.
 
I can't see any delay with ceph disks and live-migration.
 
This is migration log of VM 9006:
Code:
2017-07-19 11:10:59 migration speed: 1024.00 MB/s - downtime 12 ms
2017-07-19 11:10:59 migration status: completed
2017-07-19 11:11:00 # /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=somehost' root@1.1.1.1 pvesr set-state 9006 \''{}'\'
2017-07-19 11:11:04 migration finished successfully (duration 00:00:11)

and output of command "while true; do date; ip l show tap9006i0; sleep 1 ;done" on migration source host:
Code:
Wed Jul 19 11:10:58 MSK 2017
35: tap9006i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 02:e7:d7:63:72:c3 brd ff:ff:ff:ff:ff:ff
Wed Jul 19 11:10:59 MSK 2017
35: tap9006i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 02:e7:d7:63:72:c3 brd ff:ff:ff:ff:ff:ff
Wed Jul 19 11:11:00 MSK 2017
35: tap9006i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 02:e7:d7:63:72:c3 brd ff:ff:ff:ff:ff:ff
Wed Jul 19 11:11:01 MSK 2017
35: tap9006i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 02:e7:d7:63:72:c3 brd ff:ff:ff:ff:ff:ff
Wed Jul 19 11:11:02 MSK 2017
Device "tap9006i0" does not exist.

As you can see, migration of 9006 "finished" at 11:10:59, but device tap9006i0 disappeared at 11:11:02
 
Adding some clarification.
Presence of virtual interface in my previous post may be irrelevant to physical network, so i made network diagnostics.

I migrate VM 9006(1.1.1.111) from 1.1.1.100 to 1.1.1.102 and do arping requests from 1.1.1.101. IP and MAC addresses a not real. Here is migration log:
Code:
2017-07-19 13:30:32 start migrate command to unix:/run/qemu-server/9006.migrate
2017-07-19 13:30:34 migration status: active (transferred 362090113, remaining 3551707136), total 4312866816)
2017-07-19 13:30:34 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-19 13:30:36 migration speed: 1024.00 MB/s - downtime 22 ms
2017-07-19 13:30:36 migration status: completed
2017-07-19 13:30:37 # /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=somehost' root@1.1.1.102 pvesr set-state 9006 \''{}'\'
2017-07-19 13:30:41 migration finished successfully (duration 00:00:12)
VM migration completed at 13:30:36.

And this is tcpdump output on destination host 1.1.1.102
Code:
13:30:33.761594   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:33.761594   B ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:34.762692   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:34.762709 Out ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:34.762692   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:34.762692   B ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:35.763549   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:35.763566 Out ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:35.763549   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:35.763549   B ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:36.764546   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:36.764565 Out ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:36.764546   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:36.764546   B ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:37.765683   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:37.765704 Out ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:37.765683   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:37.765683   B ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:38.766809   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:38.766827 Out ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:38.766809   B ee:85:5a:77:77:77 ethertype 802.1Q (0x8100), length 66: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:38.766809   B ee:85:5a:77:77:77 ethertype ARP (0x0806), length 62: Request who-has 1.1.1.111 tell 1.1.1.101, length 46
13:30:39.323734   P 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.323761   P 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.323769   P 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.323775   P 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.323780   B 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Request who-has 1.1.1.111 tell 1.1.1.111, length 28
13:30:39.323786   P 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.323801   P 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.324153   B 42:86:df:88:d3:c8 ethertype 802.1Q (0x8100), length 48: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.111, length 28
13:30:39.324158   B 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Request who-has 1.1.1.111 tell 1.1.1.111, length 28
13:30:39.324333 Out 42:86:df:88:d3:c8 ethertype 802.1Q (0x8100), length 48: vlan 10, p 0, ethertype ARP, Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.324340 Out 42:86:df:88:d3:c8 ethertype 802.1Q (0x8100), length 48: vlan 10, p 0, ethertype ARP, Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.324343 Out 42:86:df:88:d3:c8 ethertype 802.1Q (0x8100), length 48: vlan 10, p 0, ethertype ARP, Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.324345 Out 42:86:df:88:d3:c8 ethertype 802.1Q (0x8100), length 48: vlan 10, p 0, ethertype ARP, Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.324351 Out 42:86:df:88:d3:c8 ethertype 802.1Q (0x8100), length 48: vlan 10, p 0, ethertype ARP, Request who-has 1.1.1.111 tell 1.1.1.111, length 28
13:30:39.324354 Out 42:86:df:88:d3:c8 ethertype 802.1Q (0x8100), length 48: vlan 10, p 0, ethertype ARP, Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.324356 Out 42:86:df:88:d3:c8 ethertype 802.1Q (0x8100), length 48: vlan 10, p 0, ethertype ARP, Reply 1.1.1.111 is-at 42:86:df:88:d3:c8, length 28
13:30:39.373192   B 42:86:df:88:d3:c8 ethertype ARP (0x0806), length 44: Request who-has 1.1.1.111 tell 1.1.1.111, length 28
The first reply is at 13:30:39.
And output of arping for completeness:
Code:
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=28 time=7.221 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=29 time=6.157 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=30 time=13.081 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=31 time=12.521 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=32 time=3.438 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=33 time=10.555 msec
Timeout
Timeout
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=34 time=575.345 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=35 time=575.377 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=36 time=575.386 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=37 time=575.393 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=38 time=575.399 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=39 time=575.405 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=40 time=575.412 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=41 time=11.016 msec
60 bytes from 42:86:df:88:d3:c8 (1.1.1.111): index=42 time=6.305 msec

So there is at minimum 2-3 second delay. Really i have done more diagnostics and problem always reproduced.

This delay may be completely unrelated to pvesr. It just introduced in same version of pve-manager and happens at the same time as pvesr command executed.
And with local storage migration this delay is 10 seconds longer.
 
You forget that you migrate a nic to a other port and the switch do not know this.
As you say you have a switch and not a hub.
So arp(ipv4) or ndp(ipv6) need some time to find the new location(port).
 
Wolfgang, i send broadcast arp requests, that go to all ports. I register all requests on destination host nic, so i'm sure that frames reach the host.
I see ARP request on nic and no ARP reply on same nic, there no switch included, except virtual one on Proxmox.
And after delay replies appeared.
 
Hello,

I can reproduce this, too. But I don't this is network related. It looks like the VM gets completely "frozen" for some seconds (here 17s).

Some details:
Empty PVE 5 test cluster with 3 nodes and 10 GBit/s network
10 GB "clean" Ubuntu VM on ZFS

Ping from outside:
Code:
PING 172.20.60.128 (172.20.60.128) 56(84) bytes of data.
..
[1500469027.533452] 64 bytes from 172.20.60.128: icmp_seq=16 ttl=64 time=0.234 ms
[1500469028.557419] 64 bytes from 172.20.60.128: icmp_seq=17 ttl=64 time=0.191 ms
[1500469045.965378] 64 bytes from 172.20.60.128: icmp_seq=34 ttl=64 time=0.148 ms
[1500469046.989439] 64 bytes from 172.20.60.128: icmp_seq=35 ttl=64 time=0.211 ms
..
^C
--- 172.20.60.128 ping statistics ---
37 packets transmitted, 21 received, 43% packet loss, time 36854ms
rtt min/avg/max/mdev = 0.133/0.213/0.831/0.142 ms
Jump from 28 to 45.

Ping from inside:
Code:
..
[1500469026.333883] 64 bytes from 8.8.8.8: icmp_seq=13 ttl=46 time=13.1 ms
[1500469027.335640] 64 bytes from 8.8.8.8: icmp_seq=14 ttl=46 time=13.4 ms
[1500469028.336981] 64 bytes from 8.8.8.8: icmp_seq=15 ttl=46 time=13.0 ms
[1500469029.338191] 64 bytes from 8.8.8.8: icmp_seq=16 ttl=46 time=13.0 ms
[1500469030.339465] 64 bytes from 8.8.8.8: icmp_seq=17 ttl=46 time=13.1 ms
..
^C
--- 8.8.8.8 ping statistics ---
21 packets transmitted, 21 received, 0% packet loss, time 20027ms
rtt min/avg/max/mdev = 13.059/13.210/13.461/0.102 ms
Freeze from 28 to 29 and 29 is 45 from the outside world..

And the VM has time skew after the migration. Hosts have the correct time.

esco
 
Confirming clock skew on migrated VM even on shared storage
Migrating VM 9004 10.0.7.149.
Code:
p@dev:[~]0$ pssh -l root -H 10.0.7.149 -H somehost01 -i date
[1] 11:46:09 [SUCCESS] somehost01
Thu Jul 20 11:46:09 MSK 2017
[2] 11:46:09 [SUCCESS] 10.0.7.149
Thu Jul 20 11:46:09 MSK 2017
p@dev:[~]0$ ssh root@somehost03 'qm migrate 9004 somehost01 --online'
2017-07-20 11:46:34 starting migration of VM 9004 to node 'somehost01' (10.0.0.100)
2017-07-20 11:46:34 copying disk images
2017-07-20 11:46:34 starting VM 9004 on remote node 'somehost01'
2017-07-20 11:46:37 start remote tunnel
2017-07-20 11:46:37 starting online/live migration on unix:/run/qemu-server/9004.migrate
2017-07-20 11:46:37 migrate_set_speed: 8589934592
2017-07-20 11:46:37 migrate_set_downtime: 0.1
2017-07-20 11:46:37 set migration_caps
2017-07-20 11:46:37 set cachesize: 429496729
2017-07-20 11:46:37 start migrate command to unix:/run/qemu-server/9004.migrate
2017-07-20 11:46:39 migration status: active (transferred 336960988, remaining 3563630592), total 4312604672)
2017-07-20 11:46:39 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2017-07-20 11:46:41 migration speed: 1024.00 MB/s - downtime 11 ms
2017-07-20 11:46:41 migration status: completed
2017-07-20 11:46:43 # /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=somehost01' root@10.0.0.100 pvesr set-state 9004 \''{}'\'
2017-07-20 11:46:46 migration finished successfully (duration 00:00:12)
p@dev:[~]0$ pssh -l root -H 10.0.7.149 -H somehost01 -i date
[1] 11:46:51 [SUCCESS] somehost01
Thu Jul 20 11:46:51 MSK 2017
[2] 11:46:51 [SUCCESS] 10.0.7.149
Thu Jul 20 11:46:48 MSK 2017
Same 3 seconds are lost
 
Last edited:
You forget that you migrate a nic to a other port and the switch do not know this.
As you say you have a switch and not a hub.
So arp(ipv4) or ndp(ipv6) need some time to find the new location(port).

Hi Wolfgang,

if guest is using virtio-net and driver have VIRTIO_NET_F_GUEST_ANNOUNCE, it should send gratuitous arp just after the live migration. Also qemu is able to send gratuitous itself if guest don't support it. So it should be fast. Maybe can we have a look at this ? (never verify when it is done exactly, after resume ?)
 
Last edited:
  • Like
Reactions: Sralityhe
Downtime improved for shared disk migration. It is now in range from 0.2 to 1 second.
For "with local disk" migration almost no effect 12.5-13 seconds.
Tested in nested cluster.

"with local disk" needs to wait for block jobs to finish, so additional delay is expected there and resuming early is dangerous. thanks for the feedback!
 
Hallo Fabian,

thank you!

Is it possible to install the package on a production system? Or, should it be done? I dont have an testenvoriment atm and dont want it to crash when i upate via your repo next time.

Kind regards
 
"with local disk" needs to wait for block jobs to finish, so additional delay is expected there and resuming early is dangerous. thanks for the feedback!

So i guess the solution atm is the best we will get in term of downtime time?
Thanks for your effort!

Kind regards
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!