Live Migration stuck at 0.0b

[jefftse]

I'm having problems to run live migration to one of my servers. It has no error message but just sitting at 0.0b. The log is below.

task started by HA resource agent
2024-02-13 10:28:29 starting migration of VM 1007 to node 'pve1' (10.0.24.8)
2024-02-13 10:28:29 starting VM 1007 on remote node 'pve1'
2024-02-13 10:28:31 start remote tunnel
2024-02-13 10:28:31 ssh tunnel ver 1
2024-02-13 10:28:31 starting online/live migration on unix:/run/qemu-server/1007.migrate
2024-02-13 10:28:31 set migration capabilities
2024-02-13 10:28:31 migration downtime limit: 100 ms
2024-02-13 10:28:31 migration cachesize: 1.0 GiB
2024-02-13 10:28:31 set migration parameters
2024-02-13 10:28:31 spice client_migrate_info
2024-02-13 10:28:31 start migrate command to unix:/run/qemu-server/1007.migrate
2024-02-13 10:28:32 migration active, transferred 2.5 MiB of 8.1 GiB VM-state, 0.0 B/s

 

[Chris]

Hi,
please check the systemd journal on both nodes for errors. You can get a paginated view of the journal since boot in reverse chronological order via journalctl -b -r. Also, please post the VM config qm config <VMID> --current as well as the VM status qm status <VMID> --verbose.

 

[jefftse]

from the source server

 

[jefftse]

qm config --current

 

[jefftse]

qm config --verbose

balloon: 8589934592
ballooninfo:
actual: 8589934592
free_mem: 5363044352
last_update: 1707840125
major_page_faults: 200787
max_mem: 8589934592
mem_swapped_in: 22237736960
mem_swapped_out: 340131840
minor_page_faults: 74528443
total_mem: 8550563840
blockstat:
efidisk0:
account_failed: 1
account_invalid: 1
failed_flush_operations: 0
failed_rd_operations: 0
failed_unmap_operations: 0
failed_wr_operations: 0
failed_zone_append_operations: 0
flush_operations: 0
flush_total_time_ns: 0
invalid_flush_operations: 0
invalid_rd_operations: 0
invalid_unmap_operations: 0
invalid_wr_operations: 0
invalid_zone_append_operations: 0
rd_bytes: 0
rd_merged: 0
rd_operations: 0
rd_total_time_ns: 0
timed_stats:
unmap_bytes: 0
unmap_merged: 0
unmap_operations: 0
unmap_total_time_ns: 0
wr_bytes: 0
wr_highest_offset: 138752
wr_merged: 0
wr_operations: 0
wr_total_time_ns: 0
zone_append_bytes: 0
zone_append_merged: 0
zone_append_operations: 0
zone_append_total_time_ns: 0
ide2:
account_failed: 1
account_invalid: 1
failed_flush_operations: 0
failed_rd_operations: 0
failed_unmap_operations: 0
failed_wr_operations: 0
failed_zone_append_operations: 0
flush_operations: 0
flush_total_time_ns: 0
invalid_flush_operations: 0
invalid_rd_operations: 0
invalid_unmap_operations: 0
invalid_wr_operations: 0
invalid_zone_append_operations: 0
rd_bytes: 0
rd_merged: 0
rd_operations: 0
rd_total_time_ns: 0
timed_stats:
unmap_bytes: 0
unmap_merged: 0
unmap_operations: 0
unmap_total_time_ns: 0
wr_bytes: 0
wr_highest_offset: 0
wr_merged: 0
wr_operations: 0
wr_total_time_ns: 0
zone_append_bytes: 0
zone_append_merged: 0
zone_append_operations: 0
zone_append_total_time_ns: 0
pflash0:
account_failed: 1
account_invalid: 1
failed_flush_operations: 0
failed_rd_operations: 0
failed_unmap_operations: 0
failed_wr_operations: 0
failed_zone_append_operations: 0
flush_operations: 0
flush_total_time_ns: 0
invalid_flush_operations: 0
invalid_rd_operations: 0
invalid_unmap_operations: 0
invalid_wr_operations: 0
invalid_zone_append_operations: 0
rd_bytes: 0
rd_merged: 0
rd_operations: 0
rd_total_time_ns: 0
timed_stats:
unmap_bytes: 0
unmap_merged: 0
unmap_operations: 0
unmap_total_time_ns: 0
wr_bytes: 0
wr_highest_offset: 0
wr_merged: 0
wr_operations: 0
wr_total_time_ns: 0
zone_append_bytes: 0
zone_append_merged: 0
zone_append_operations: 0
zone_append_total_time_ns: 0
virtio0:
account_failed: 1
account_invalid: 1
failed_flush_operations: 0
failed_rd_operations: 0
failed_unmap_operations: 0
failed_wr_operations: 0
failed_zone_append_operations: 0
flush_operations: 40995
flush_total_time_ns: 62511539157
idle_time_ns: 2724357335
invalid_flush_operations: 0
invalid_rd_operations: 0
invalid_unmap_operations: 0
invalid_wr_operations: 0
invalid_zone_append_operations: 0
rd_bytes: 22821671424
rd_merged: 6061
rd_operations: 195421
rd_total_time_ns: 277940198022
timed_stats:
unmap_bytes: 0
unmap_merged: 0
unmap_operations: 0
unmap_total_time_ns: 0
wr_bytes: 19645223936
wr_highest_offset: 321649332224
wr_merged: 3725
wr_operations: 274583
wr_total_time_ns: 3883214557217
zone_append_bytes: 0
zone_append_merged: 0
zone_append_operations: 0
zone_append_total_time_ns: 0
cpus: 6
disk: 0
diskread: 22821671424
diskwrite: 19645223936
freemem: 5363044352
lock: migrate
maxdisk: 322122547200
maxmem: 8589934592
mem: 3187519488
name: Jeff-VM01
netin: 18789229138
netout: 15832949344
nics:
tap1008i0:
netin: 18789229138
netout: 15832949344
pid: 202327
proxmox-support:
backup-max-workers: 1
pbs-dirty-bitmap: 1
pbs-dirty-bitmap-migration: 1
pbs-dirty-bitmap-savevm: 1
pbs-library-version: 1.4.1 (UNKNOWN)
pbs-masterkey: 1
query-bitmap-info: 1
qmpstatus: running
running-machine: pc-q35-8.1+pve0
running-qemu: 8.1.5
status: running
uptime: 65763
vmid: 1008

 

[Chris]

Is the migration network fine? The broken pipe might indicate that this was not the case. Please try to stop the migration, check the migration network connectivity and restart the migration.

 

[jefftse]

this is a 6 node cluster and I have no problem running live migration to any other server but that and they all have the identical setup

 

[Chris]

So if only one node is affected, then there is probably something off with that one. Please check the systemd journal on that target node during the migration timespan and check if the target node has the required resources, e.g. storage space, ram, ecc and network connectivity. Does this node have different hardware? Double check also the pveversion -v for possible differences.

 

[fiona]

from the source server
View attachment 63082

In the screenshot, the error

kvm: failed to save SaveStateEntry with id(name): 3(ram): -5

can be seen. Now, the error code is

febner@enia ~ % errno 5
EIO 5 Input/output error

From this, the best guess is that there was an IO error when migration tried to transfer part of the RAM. As @Chris suggested, please check your network connection.