Life Migration with local storage: 3-9 seconds downtime (qemu reports 100ms ...)

[HeB]

Hello,
when I life migrate small debian VMs between nodes with local storage the VM finally is unavailable for 3-9s.
After the migration NTP time synchronisation shows an offset o f 3-9s, too.
The VM is small (4GB, 25GB ) and idle. The nodes have little load and have 2x25GbE. The Migration itself takes about 15s, but the final switch, where the VM is unavailable takes a few seconds. I did expect some 100ms as qemu reports in the migration task log.

Please
- are there known changes to try?
- how can I get a more detailed log for. the migration (qemu and pve part)

Thank you,

Heiner



the migration task show this logs


2026-06-29 11:33:18 starting migration of VM 1300 to node 'nodec' (x.y.x.x)
2026-06-29 11:33:18 found local, replicated disk 'pvepool:vm-1300-disk-0' (attached)
2026-06-29 11:33:18 scsi0: start tracking writes using block-dirty-bitmap 'repl_scsi0'
2026-06-29 11:33:18 replicating disk images
2026-06-29 11:33:18 start replication job
skipping guest filesystem freeze - disabled in VM options
2026-06-29 11:33:18 guest => VM 1300, running => 23900
2026-06-29 11:33:18 volumes => pvepool:vm-1300-disk-0
2026-06-29 11:33:19 create snapshot '__replicate_1300-1_1782725598__' on pvepool:vm-1300-disk-0
2026-06-29 11:33:19 using secure transmission, rate limit: none
2026-06-29 11:33:19 incremental sync 'pvepool:vm-1300-disk-0' (__replicate_1300-1_1782723605__ => __replicate_1300-1_1782725598__)
2026-06-29 11:33:19 successfully imported 'pvepool:vm-1300-disk-0'
2026-06-29 11:33:19 send from @__replicate_1300-1_1782723605__ to pvepool/vm-1300-disk-0@__replicate_1300-1_1782725598__ estimated size is 11.5Mtotal estimated size is 11.5MTIME SENT SNAPSHOT pvepool/vm-1300-disk-0@__replicate_1300-1_1782725598__
2026-06-29 11:33:19 delete previous replication snapshot '__replicate_1300-1_1782723605__' on pvepool:vm-1300-disk-0
2026-06-29 11:33:20 (remote_finalize_local_job) delete stale replication snapshot '__replicate_1300-1_1782723605__' on pvepool:vm-1300-disk-0
2026-06-29 11:33:20 end replication job
2026-06-29 11:33:20 starting VM 1300 on remote node 'nodec'
2026-06-29 11:33:21 volume 'pvepool:vm-1300-disk-0' is 'pvepool:vm-1300-disk-0' on the target
2026-06-29 11:33:21 start remote tunnel
2026-06-29 11:33:22 ssh tunnel ver 1
2026-06-29 11:33:22 starting storage migration
2026-06-29 11:33:22 scsi0: start migration to nbd:unix:/run/qemu-server/1300_nbd.migrate:exportname=drive-scsi0
drive mirror re-using dirty bitmap 'repl_scsi0'
drive mirror is starting for drive-scsi0
mirror-scsi0: transferred 0.0 B of 128.0 KiB (0.00%) in 0s
mirror-scsi0: transferred 128.0 KiB of 128.0 KiB (100.00%) in 1s, ready
all 'mirror' jobs are ready
2026-06-29 11:33:23 switching mirror jobs to actively synced mode
mirror-scsi0: switching to actively synced mode
mirror-scsi0: successfully switched to actively synced mode
2026-06-29 11:33:24 starting online/live migration on unix:/run/qemu-server/1300.migrate
2026-06-29 11:33:24 set migration capabilities
2026-06-29 11:33:24 migration downtime limit: 100 ms
2026-06-29 11:33:24 migration cachesize: 512.0 MiB
2026-06-29 11:33:24 set migration parameters
This is the critical part of the log, Please note: qemu reports 92ms downtime only.
I don't see a single entry to take. 3.5s?
does the disk take so long to become active?

2026-06-29 11:33:24 start migrate command to unix:/run/qemu-server/1300.migrate
2026-06-29 11:33:25 migration active, transferred 825.1 MiB of 4.0 GiB VM-state, 1.2 GiB/s
2026-06-29 11:33:26 average migration speed: 2.0 GiB/s - downtime 92 ms
2026-06-29 11:33:26 migration completed, transferred 1.4 GiB VM-state
2026-06-29 11:33:26 migration status: completed
all 'mirror' jobs are ready
mirror-scsi0: Completing block job...
mirror-scsi0: Completed successfully.
mirror-scsi0: mirror-job finished
2026-06-29 11:33:28 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=nodec' -o 'UserKnownHostsFile=/etc/pve/nodes/nodec/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' root@x.x.x.x pvesr set-state 1300 \''{"local/nodea":{"last_try":1782725598,"storeid_list":["pvepool"],"last_node":"node01","last_iteration":1782725598,"duration":1.721744,"fail_count":0,"last_sync":1782725598}}'\'
2026-06-29 11:33:28 stopping NBD storage migration server on target.
2026-06-29 11:33:29 stopping migration dbus-vmstate helpers
2026-06-29 11:33:29 migrated 0 conntrack state entries

2026-06-29 11:33:30 flushing conntrack state for guest on source node
2026-06-29 11:33:32 migration finished successfully (duration 00:00:15)
TASK OK
```


ntpdate s shows this time offset. I did call it every second.The VM was unresponsive at 08:33:25-29
2026-06-29 09:33:21.238151 (+0000) -0.001857 +/- 0.000133 10.1.10.54 s2 no-leap
2026-06-29 09:33:22.266540 (+0000) -0.001861 +/- 0.000120 10.1.10.54 s2 no-leap
2026-06-29 09:33:23.294222 (+0000) -0.001863 +/- 0.000114 10.1.10.54 s2 no-leap
2026-06-29 09:33:24.321916 (+0000) -0.001868 +/- 0.000109 10.1.10.54 s2 no-leap
2026-06-29 [B]09:33:25.351398[/B] (+0000) -0.001866 +/- 0.000118 10.1.10.54 s2 no-leap still in sync
2026-06-29 [B]09:33:29.951654[/B] (+0000) [B]+3.569573 [/B]+/- 0.000139 10.1.10.54 s2 no-leap lost 3.5 seconds
2026-06-29 09:33:30.977948 (+0000) +3.569582 +/- 0.000127 10.1.10.54 s2 no-leap
2026-06-29 09:33:32.004644 (+0000) +3.569578 +/- 0.000136 10.1.10.54 s2 no-leap
2026-06-29 09:33:33.030686 (+0000) +3.569566 +/- 0.000119 10.1.10.54 s2 no-leap
2026-06-29 09:33:34.056322 (+0000) +3.569569 +/- 0.000118 10.1.10.54 s2 no-leap
2026-06-29 09:33:35.081865 (+0000) +3.569559 +/- 0.000120 10.1.10.54 s2 no-leap

Versions

proxmox-ve: 9.2.0 (running kernel: 7.0.12-1-pve)
pve-manager: 9.2.3 (running version: 9.2.3/d0fde103346cf89a)
Debian installation



corosync: 3.1.10-pve2

 

[Alwin Antreich]

Is the target node loaded? Look at the pressure stall values, anything > 0?

 

[fiona]

Hi,
if you use local storage, then the downtime will be longer. The downtime that is currently logged in the task log is only for the RAM/VM state migration itself. But the VM can only be resumed after the storage migration is fully finished, i.e. after stopping the NBD server on the target. @jklocker is currently working on bug #5032 to automatically sync the clock afterwards. If you require less downtime, using a shared storage should mitigate the issue.