[SOLVED] PVE 5 Live migration downtime degradation (2-4 sec)

P

Petr

Member
Dec 13, 2013
40
1
6
The problem found in thread https://forum.proxmox.com/threads/slow-livemigration-performance-since-5-0.35522/, but there discussed live migration with local disks. To not hijack that thread, I open new one.

In PVE 5.0 after live migration with shared storage, VM hangs for 2-4 seconds. It can be checked by comaring localtime before and after migration. I setted up 3 nested clusters with PVE 4.4, 5.0 beta2 and 5.0.

For test executing date command in parallel-ssh before and after migration, comparing output on PVE nodes and migrated VM. Both 5 versions have a 2-4 seconds skew. In 4.4 clock skew is very small, much less then 1 second.
Code: 
root@nat01:~# parallel-ssh -H 10.10.10.202 -H 10.10.10.201 -i pveversion
[1] 18:11:39 [SUCCESS] 10.10.10.201
pve-manager/5.0-23/af4267bf (running kernel: 4.10.15-1-pve)
[2] 18:11:40 [SUCCESS] 10.10.10.202
pve-manager/5.0-23/af4267bf (running kernel: 4.10.15-1-pve)
root@nat01:~# parallel-ssh -H 10.10.10.202 -H 172.17.2.108 -H 10.10.10.201 -i date
[1] 18:11:42 [SUCCESS] 172.17.2.108
Mon Jul 24 18:11:42 MSK 2017
[2] 18:11:42 [SUCCESS] 10.10.10.202
Mon Jul 24 18:11:42 MSK 2017
[3] 18:11:42 [SUCCESS] 10.10.10.201
Mon Jul 24 18:11:42 MSK 2017
root@nat01:~# ssh 10.10.10.201 qm migrate 300 pve02 --online
2017-07-24 18:11:46 starting migration of VM 300 to node 'pve02' (10.10.10.202)
2017-07-24 18:11:46 copying disk images
2017-07-24 18:11:46 starting VM 300 on remote node 'pve02'
2017-07-24 18:11:49 start remote tunnel
2017-07-24 18:11:49 starting online/live migration on unix:/run/qemu-server/300.migrate
2017-07-24 18:11:49 migrate_set_speed: 8589934592
2017-07-24 18:11:49 migrate_set_downtime: 0.1
2017-07-24 18:11:49 set migration_caps
2017-07-24 18:11:49 set cachesize: 53687091
2017-07-24 18:11:49 start migrate command to unix:/run/qemu-server/300.migrate
2017-07-24 18:11:51 migration speed: 256.00 MB/s - downtime 14 ms
2017-07-24 18:11:51 migration status: completed
2017-07-24 18:11:52 # /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=pve02' root@10.10.10.202 pvesr set-state 300 \''{}'\'
2017-07-24 18:11:56 migration finished successfully (duration 00:00:10)
root@nat01:~# parallel-ssh -H 10.10.10.202 -H 172.17.2.108 -H 10.10.10.201 -i date
[1] 18:12:03 [SUCCESS] 10.10.10.202
Mon Jul 24 18:12:03 MSK 2017
[2] 18:12:03 [SUCCESS] 172.17.2.108
Mon Jul 24 18:11:59 MSK 2017
[3] 18:12:03 [SUCCESS] 10.10.10.201
Mon Jul 24 18:12:03 MSK 2017
4 seconds in PVE 5.0
Code: 
root@nat01:~# parallel-ssh -H 10.10.10.203 -H 10.10.10.204 -i pveversion
[1] 18:12:21 [SUCCESS] 10.10.10.203
pve-manager/5.0-10/0d270679 (running kernel: 4.10.11-1-pve)
[2] 18:12:21 [SUCCESS] 10.10.10.204
pve-manager/5.0-10/0d270679 (running kernel: 4.10.11-1-pve)
root@nat01:~# parallel-ssh -H 10.10.10.203 -H 172.17.2.178 -H 10.10.10.204 -i date
[1] 18:12:26 [SUCCESS] 10.10.10.204
Mon Jul 24 18:12:26 MSK 2017
[2] 18:12:26 [SUCCESS] 10.10.10.203
Mon Jul 24 18:12:26 MSK 2017
[3] 18:12:26 [SUCCESS] 172.17.2.178
Mon Jul 24 18:12:26 MSK 2017
root@nat01:~# ssh 10.10.10.203 qm migrate 100 pve04 --online
Jul 24 18:12:33 starting migration of VM 100 to node 'pve04' (10.10.10.204)
Jul 24 18:12:33 copying disk images
Jul 24 18:12:33 starting VM 100 on remote node 'pve04'
Jul 24 18:12:36 start remote tunnel
Jul 24 18:12:37 starting online/live migration on unix:/run/qemu-server/100.migrate
Jul 24 18:12:37 migrate_set_speed: 8589934592
Jul 24 18:12:37 migrate_set_downtime: 0.1
Jul 24 18:12:37 set migration_caps
Jul 24 18:12:37 set cachesize: 53687091
Jul 24 18:12:37 start migrate command to unix:/run/qemu-server/100.migrate
Jul 24 18:12:39 migration speed: 256.00 MB/s - downtime 39 ms
Jul 24 18:12:39 migration status: completed
Jul 24 18:12:43 migration finished successfully (duration 00:00:10)
root@nat01:~# parallel-ssh -H 10.10.10.203 -H 172.17.2.178 -H 10.10.10.204 -i date
[1] 18:12:48 [SUCCESS] 172.17.2.178
Mon Jul 24 18:12:45 MSK 2017
[2] 18:12:48 [SUCCESS] 10.10.10.204
Mon Jul 24 18:12:48 MSK 2017
[3] 18:12:48 [SUCCESS] 10.10.10.203
Mon Jul 24 18:12:48 MSK 2017
3 seconds in PVE 5.0 beta2
Code: 
root@nat01:~# parallel-ssh -H 10.10.10.205 -H 10.10.10.206 -i pveversion
[1] 18:12:58 [SUCCESS] 10.10.10.206
pve-manager/4.4-1/eb2d6f1e (running kernel: 4.4.35-1-pve)
[2] 18:12:58 [SUCCESS] 10.10.10.205
pve-manager/4.4-1/eb2d6f1e (running kernel: 4.4.35-1-pve)
root@nat01:~# parallel-ssh -H 10.10.10.206 -H 172.17.2.247 -H 10.10.10.205 -i date
[1] 18:13:01 [SUCCESS] 10.10.10.206
Mon Jul 24 18:13:01 MSK 2017
[2] 18:13:01 [SUCCESS] 10.10.10.205
Mon Jul 24 18:13:01 MSK 2017
[3] 18:13:01 [SUCCESS] 172.17.2.247
Mon Jul 24 18:13:01 MSK 2017
root@nat01:~# ssh 10.10.10.205 qm migrate 200 pve06 --online
Jul 24 18:13:05 starting migration of VM 200 to node 'pve06' (10.10.10.206)
Jul 24 18:13:05 copying disk images
Jul 24 18:13:05 starting VM 200 on remote node 'pve06'
Jul 24 18:13:08 start remote tunnel
Jul 24 18:13:09 starting online/live migration on unix:/run/qemu-server/200.migrate
Jul 24 18:13:09 migrate_set_speed: 8589934592
Jul 24 18:13:09 migrate_set_downtime: 0.1
Jul 24 18:13:09 set migration_caps
Jul 24 18:13:09 set cachesize: 53687091
Jul 24 18:13:09 start migrate command to unix:/run/qemu-server/200.migrate
Jul 24 18:13:11 migration speed: 256.00 MB/s - downtime 16 ms
Jul 24 18:13:11 migration status: completed
Jul 24 18:13:15 migration finished successfully (duration 00:00:11)
root@nat01:~# parallel-ssh -H 10.10.10.206 -H 172.17.2.247 -H 10.10.10.205 -i date
[1] 18:13:22 [SUCCESS] 10.10.10.205
Mon Jul 24 18:13:22 MSK 2017
[2] 18:13:22 [SUCCESS] 10.10.10.206
Mon Jul 24 18:13:22 MSK 2017
[3] 18:13:22 [SUCCESS] 172.17.2.247
Mon Jul 24 18:13:22 MSK 2017
less then 1 second in PVE 4.4

IMO 3 seconds is gigantic delay and no-go for some production systems.
 
I have no issue on my cluster this is lightning fast :)
all on shared ceph storage, no zfs involved.

Code: 
task started by HA resource agent
2017-07-25 15:00:53 starting migration of VM 101 to node 'pve03' (192.168.221.143)
2017-07-25 15:00:53 copying disk images
2017-07-25 15:00:53 starting VM 101 on remote node 'pve03'
2017-07-25 15:00:55 start remote tunnel
2017-07-25 15:00:55 starting online/live migration on unix:/run/qemu-server/101.migrate
2017-07-25 15:00:55 migrate_set_speed: 8589934592
2017-07-25 15:00:55 migrate_set_downtime: 0.1
2017-07-25 15:00:55 set migration_caps
2017-07-25 15:00:55 set cachesize: 848927129
2017-07-25 15:00:55 start migrate command to unix:/run/qemu-server/101.migrate
2017-07-25 15:00:57 migration speed: 4048.00 MB/s - downtime 46 ms
2017-07-25 15:00:57 migration status: completed
2017-07-25 15:00:58 # /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=pve03' root@192.168.221.143 pvesr set-state 101 \''{}'\'
2017-07-25 15:01:01 migration finished successfully (duration 00:00:08)
TASK OK
 
Gerhard, can you please check localtime or ntp offset inside VM before and after migration, or how many ICMP pings to VM was lost during migration?
 
Petr said:
Gerhard, can you please check localtime or ntp offset inside VM before and after migration, or how many ICMP pings to VM was lost during migration?
Click to expand...

icmp from a cluster host to vm
Code: 
 ping 192.168.221.151
PING 192.168.221.151 (192.168.221.151) 56(84) bytes of data.
64 bytes from 192.168.221.151: icmp_seq=1 ttl=128 time=0.232 ms
64 bytes from 192.168.221.151: icmp_seq=2 ttl=128 time=0.224 ms
64 bytes from 192.168.221.151: icmp_seq=3 ttl=128 time=0.343 ms
64 bytes from 192.168.221.151: icmp_seq=4 ttl=128 time=0.280 ms
64 bytes from 192.168.221.151: icmp_seq=5 ttl=128 time=0.136 ms
64 bytes from 192.168.221.151: icmp_seq=6 ttl=128 time=0.659 ms
64 bytes from 192.168.221.151: icmp_seq=7 ttl=128 time=0.212 ms
64 bytes from 192.168.221.151: icmp_seq=8 ttl=128 time=0.378 ms
64 bytes from 192.168.221.151: icmp_seq=9 ttl=128 time=0.377 ms
64 bytes from 192.168.221.151: icmp_seq=10 ttl=128 time=0.157 ms
64 bytes from 192.168.221.151: icmp_seq=11 ttl=128 time=0.209 ms
64 bytes from 192.168.221.151: icmp_seq=12 ttl=128 time=0.236 ms
64 bytes from 192.168.221.151: icmp_seq=13 ttl=128 time=0.214 ms
64 bytes from 192.168.221.151: icmp_seq=14 ttl=128 time=0.188 ms
64 bytes from 192.168.221.151: icmp_seq=15 ttl=128 time=0.199 ms
64 bytes from 192.168.221.151: icmp_seq=16 ttl=128 time=0.204 ms
64 bytes from 192.168.221.151: icmp_seq=17 ttl=128 time=0.223 ms
64 bytes from 192.168.221.151: icmp_seq=18 ttl=128 time=0.178 ms
64 bytes from 192.168.221.151: icmp_seq=19 ttl=128 time=0.139 ms
64 bytes from 192.168.221.151: icmp_seq=20 ttl=128 time=0.169 ms
64 bytes from 192.168.221.151: icmp_seq=21 ttl=128 time=0.155 ms
64 bytes from 192.168.221.151: icmp_seq=22 ttl=128 time=0.324 ms
64 bytes from 192.168.221.151: icmp_seq=23 ttl=128 time=0.333 ms
64 bytes from 192.168.221.151: icmp_seq=24 ttl=128 time=0.243 ms
64 bytes from 192.168.221.151: icmp_seq=25 ttl=128 time=0.374 ms
64 bytes from 192.168.221.151: icmp_seq=26 ttl=128 time=0.266 ms
64 bytes from 192.168.221.151: icmp_seq=29 ttl=128 time=0.224 ms
64 bytes from 192.168.221.151: icmp_seq=30 ttl=128 time=0.363 ms
64 bytes from 192.168.221.151: icmp_seq=31 ttl=128 time=0.288 ms
64 bytes from 192.168.221.151: icmp_seq=32 ttl=128 time=0.326 ms
64 bytes from 192.168.221.151: icmp_seq=33 ttl=128 time=0.377 ms
64 bytes from 192.168.221.151: icmp_seq=34 ttl=128 time=0.202 ms
64 bytes from 192.168.221.151: icmp_seq=35 ttl=128 time=0.173 ms
64 bytes from 192.168.221.151: icmp_seq=36 ttl=128 time=0.133 ms
64 bytes from 192.168.221.151: icmp_seq=37 ttl=128 time=0.254 ms
64 bytes from 192.168.221.151: icmp_seq=38 ttl=128 time=0.213 ms
64 bytes from 192.168.221.151: icmp_seq=39 ttl=128 time=0.339 ms
64 bytes from 192.168.221.151: icmp_seq=40 ttl=128 time=0.347 ms
64 bytes from 192.168.221.151: icmp_seq=41 ttl=128 time=0.324 ms
64 bytes from 192.168.221.151: icmp_seq=42 ttl=128 time=0.374 ms
64 bytes from 192.168.221.151: icmp_seq=43 ttl=128 time=0.289 ms
64 bytes from 192.168.221.151: icmp_seq=44 ttl=128 time=0.212 ms
64 bytes from 192.168.221.151: icmp_seq=45 ttl=128 time=0.256 ms
64 bytes from 192.168.221.151: icmp_seq=46 ttl=128 time=0.226 ms
64 bytes from 192.168.221.151: icmp_seq=47 ttl=128 time=0.243 ms
^C
--- 192.168.221.151 ping statistics ---
47 packets transmitted, 45 received, 4% packet loss, time 47095ms
rtt min/avg/max/mdev = 0.133/0.262/0.659/0.096 ms
root@pve04:~#

and inside vm to csame clusterhost
Code: 
C:\Users\Administrator>ping 192.168.221.144 -t

Ping wird ausgeführt für 192.168.221.144 mit 32 Bytes Daten:
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Zeitüberschreitung der Anforderung.
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64
Antwort von 192.168.221.144: Bytes=32 Zeit<1ms TTL=64

Ping-Statistik für 192.168.221.144:
    Pakete: Gesendet = 60, Empfangen = 59, Verloren = 1
    (1% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 0ms, Maximum = 0ms, Mittelwert = 0ms
STRG-C
^C
C:\Users\Administrator>
Code: 
ask started by HA resource agent
2017-07-25 16:14:54 starting migration of VM 101 to node 'pve02' (192.168.221.142)
2017-07-25 16:14:54 copying disk images
2017-07-25 16:14:54 starting VM 101 on remote node 'pve02'
2017-07-25 16:14:56 start remote tunnel
2017-07-25 16:14:56 starting online/live migration on unix:/run/qemu-server/101.migrate
2017-07-25 16:14:56 migrate_set_speed: 8589934592
2017-07-25 16:14:56 migrate_set_downtime: 0.1
2017-07-25 16:14:56 set migration_caps
2017-07-25 16:14:56 set cachesize: 848927129
2017-07-25 16:14:56 start migrate command to unix:/run/qemu-server/101.migrate
2017-07-25 16:14:58 migration speed: 4048.00 MB/s - downtime 40 ms
2017-07-25 16:14:58 migration status: completed
2017-07-25 16:14:59 # /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=pve02' root@192.168.221.142 pvesr set-state 101 \''{}'\'
2017-07-25 16:15:02 migration finished successfully (duration 00:00:08)
TASK OK
 
You have 2 packets lost:
47 packets transmitted, 45 received, 4% packet loss, time 47095ms
rtt min/avg/max/mdev = 0.133/0.262/0.659/0.096 ms
Click to expand...
in my cluster i have 4 ping lost, but 2 is too much anyway.

If you try same experiment on PVE 4, there will be maximum 1 ping lost. Most of a time no loss at all.
I now have 2 nested clusters with identical configuration, but different versions.

This is stats in PVE 4 during migration:
24 packets transmitted, 24 received, 0% packet loss, time 23006ms
rtt min/avg/max/mdev = 0.308/0.967/3.380/0.943 ms

And the same configuration, but PVE 5:
18 packets transmitted, 14 received, 22% packet loss, time 17311ms
rtt min/avg/max/mdev = 0.272/1.440/10.092/2.470 ms
 
Petr said:
You have 2 packets lost:

in my cluster i have 4 ping lost, but 2 is too much anyway.

If you try same experiment on PVE 4, there will be maximum 1 ping lost. Most of a time no loss at all.
I now have 2 nested clusters with identical configuration, but different versions.

This is stats in PVE 4 during migration:
24 packets transmitted, 24 received, 0% packet loss, time 23006ms
rtt min/avg/max/mdev = 0.308/0.967/3.380/0.943 ms

And the same configuration, but PVE 5:
18 packets transmitted, 14 received, 22% packet loss, time 17311ms
rtt min/avg/max/mdev = 0.272/1.440/10.092/2.470 ms
Click to expand...


Petr, perhaps this is a arp problem within migration task ? would be nice someone from staff came across with a explanation :(
 
Gerhard, unfortunately this is not just network problem, but VM is really frozen for some seconds insted of few dozens of milliseconds. Ping is just simple way to demonstrate the bug.
 
spirit said:
can you try to add in /etc/pve/datacenter.cfg

migration: type=insecure

to disable the ssh tunnel, and try migration again ?
Click to expand...
i've already tryed setting insecure option in migration parameters and did not find noticeable difference
 
spirit, I had to get my hands dirty with nice PVE perl code and not so nice qemu qmp sokcets protocol and mostly found сause of the problem.
On target host VM starts in paused state.
Then, in version 4 after migration process finishes, VM state transfers from '"VM status: paused (inmigrate)' to '"VM status: running' on target host.
In version 5 status changes to '"VM status: paused' and waits for command 'qm resume' from QemuMigrate.pm module. Same command executed on PVE 4 too, but at that time VM is already running for 2 seconds.
Now is not completely clear why this behavior changed. I'm trying to find it out.
If you interested, i can tell you how to verify this issue.
 
proxmox 4 or 5, the vm is always pause on target during the migration, and resume after the live migration + some checks (maybe the problem is here)

I think new code is:

/usr/share/perl5/PVE/QemuMigrate.pm
sub phase3_cleanup {
...

# transfer replication state before move config
$self->transfer_replication_state();
...
$self->switch_replication_job_target();

....
if ($self->{storage_migration}) {
# remove drives referencing the nbd server from source
# otherwise vm_stop might hang later on
foreach my $drive (keys %{$self->{target_drive}}){
PVE::QemuServer::vm_mon_cmd_nocheck($vmid, "device_del", id => $drive);
}
# stop nbd server on remote vm - requirement for resume since 2.9
my $cmd = [@{$self->{rem_ssh}}, 'qm', 'nbdstop', $vmid];

eval{ PVE::Tools::run_command($cmd, outfunc => sub {}, errfunc => sub {}) };
if (my $err = $@) {
$self->log('err', $err);
$self->{errors} = 1;
}
}




So , maybe if related to replication_job (it shouldn't be use).
Maybe try to comment code on both source and dest, and restart pvedaemon service.

If it's don't change anything, maybe it's related to qemu 2.9 ....
 
I also thought that it would be that easy =).
But i already tried to comment out storage checks and already upgraded pve-qemu-kvm to 2.9 on PVE 4. No result.
Adding 'qm resume' command to QemuMigrate.pm right after migration in "phase 2" helps, but not completely. 'qm' tool is not fast enough and 0.5-1 second freeze remains.
The only ugly hack that allows me to migrate without downtime in PVE 5 is raw 'cont' command to the qmp socket (https://en.wikibooks.org/wiki/QEMU/Monitor#cont).
Will continue experiments tomorrow if Proxmox staff not get interested with this topic.
 
Petr said:
I also thought that it would be that easy =).
But i already tried to comment out storage checks and already upgraded pve-qemu-kvm to 2.9 on PVE 4. No result.
Adding 'qm resume' command to QemuMigrate.pm right after migration in "phase 2" helps, but not completely. 'qm' tool is not fast enough and 0.5-1 second freeze remains.
The only ugly hack that allows me to migrate without downtime in PVE 5 is raw 'cont' command to the qmp socket (https://en.wikibooks.org/wiki/QEMU/Monitor#cont).
Will continue experiments tomorrow if Proxmox staff not get interested with this topic.
Click to expand...

pretty strange...

qm resume only connect to ssh on remote target and call

/usr/share/perl5/PVE/QemuServer.pm

sub vm_resume {
my ($vmid, $skiplock, $nocheck) = @_;

PVE::QemuConfig->lock_config($vmid, sub {

if (!$nocheck) {

my $conf = PVE::QemuConfig->load_config($vmid);

PVE::QemuConfig->check_lock($conf)
if !($skiplock || PVE::QemuConfig->has_lock($conf, 'backup'));

vm_mon_cmd($vmid, "cont");

} else {
vm_mon_cmd_nocheck($vmid, "cont");
}
});
}
 
spirit said:
pretty strange...

qm resume only connect to ssh on remote target and call

/usr/share/perl5/PVE/QemuServer.pm

sub vm_resume {
my ($vmid, $skiplock, $nocheck) = @_;

PVE::QemuConfig->lock_config($vmid, sub {

if (!$nocheck) {

my $conf = PVE::QemuConfig->load_config($vmid);

PVE::QemuConfig->check_lock($conf)
if !($skiplock || PVE::QemuConfig->has_lock($conf, 'backup'));

vm_mon_cmd($vmid, "cont");

} else {
vm_mon_cmd_nocheck($vmid, "cont");
}
});
}
Click to expand...
yes, but this process takes some time, about halfsecond. As i said, In PVE 4 VM becomes running before this command executed. And it is impossible to get downtime ~20 ms if you are waiting for such commands.
This is why i think it is obvious bug.
Now to emulate behavior of PVE 4, i send 'cont' in the loop in bash script on target host.
 
Petr said:
As i said, In PVE 4 VM becomes running before this command executed. t.
Click to expand...
I'll test it, but this is strange. vm should be paused until proxmox resume it.


another thing to try,

another change in proxmox5
Code: 
--- a/PVE/QemuMigrate.pm
+++ b/PVE/QemuMigrate.pm
@@ -96,7 +96,7 @@ sub fork_tunnel {

     my @localtunnelinfo = defined($tunnel_addr) ? ('-L' , $tunnel_addr ) : ();

-    my $cmd = [@{$self->{rem_ssh}}, '-o ExitOnForwardFailure=yes', @localtunnelinfo, 'qm', 'mtunnel' ];
+    my $cmd = [@{$self->{rem_ssh}}, '-o ExitOnForwardFailure=yes', @localtunnelinfo, '/usr/bin/pvecm', 'mtunnel' ];

     my $tunnel = $self->fork_command_pipe($cmd);

maybe try to revert to qm tunnel, to see the difference.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom