[SOLVED] Live migration error (GUI and CLI)

[Frosty]

2019-12-16 13:59:25 starting migration of VM 111 to node 'bigv2v-01' (xx.xx.xx.xx)
2019-12-16 13:59:25 found local disk 'bigbertha-data:vm-111-disk-0' (in current VM config)
2019-12-16 13:59:25 copying local disk images
2019-12-16 13:59:25 starting VM 111 on remote node 'bigv2v-01'
2019-12-16 13:59:28 start remote tunnel
2019-12-16 13:59:28 ssh tunnel ver 1
2019-12-16 13:59:28 starting storage migration
2019-12-16 13:59:28 virtio0: start migration to nbd:xx.xx.xx.xx:60000:exportname=drive-virtio0
drive mirror is starting for drive-virtio0
drive-virtio0: Cancelling block job
drive-virtio0: Done.
2019-12-16 14:01:39 ERROR: online migrate failure - mirroring error: VM 111 qmp command 'drive-mirror' failed - got timeout
2019-12-16 14:01:39 aborting phase 2 - cleanup resources
2019-12-16 14:01:39 migrate_cancel
2019-12-16 14:01:47 ERROR: migration finished with problems (duration 00:02:22)
TASK ERROR: migration problems

Node 1: pveversion -v

Spoiler

proxmox-ve: 6.1-2 (running kernel: 5.3.10-1-pve)
pve-manager: 6.1-3 (running version: 6.1-3/37248ce6)
pve-kernel-5.3: 6.0-12
pve-kernel-helper: 6.0-12
pve-kernel-4.15: 5.4-11
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-17-pve: 4.15.18-43
pve-kernel-4.15.18-16-pve: 4.15.18-41
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 1.2.8-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-14
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-2
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

node 2: pveversion -v

Spoiler

proxmox-ve: 6.1-2 (running kernel: 5.3.10-1-pve)
pve-manager: 6.1-3 (running version: 6.1-3/37248ce6)
pve-kernel-5.3: 6.0-12
pve-kernel-helper: 6.0-12
pve-kernel-4.15: 5.4-11
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-4.15.18-23-pve: 4.15.18-51
ceph: 14.2.4.1-pve1
ceph-fuse: 14.2.4.1-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 1.2.8-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-14
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.0-9[/spoiler-box]
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-2
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

cat /etc/pve/storage.cfg

Spoiler

dir: bigbertha
path /bigbertha/data
content iso,images,vztmpl,rootdir,backup,snippets
maxfiles 0
nodes bigbertha
shared 0

dir: local
path /var/lib/vz
content iso,vztmpl,images,rootdir,snippets
maxfiles 0

zfspool: rpool-data
pool rpool/data
blocksize 4K
content rootdir,images
nodes bigbertha
sparse 0

zfspool: bigbertha-data
pool bigbertha/data
blocksize 4k
content images,rootdir
nodes bigbertha
sparse 0

dir: rpool
path /rpool/data
content images,vztmpl,iso,backup,snippets,rootdir
maxfiles 1
nodes bigv2v-01
shared 0

zfspool: bigv2v-01-data
pool bigv2v-01-data
blocksize 4k
content images,rootdir
mountpoint /bigv2v-01-data
nodes bigv2v-01
sparse 1

nfs: rpool-sharenfs
export /rpool/data
path /mnt/pve/rpool-sharenfs
server 10.1.0.2
content iso,rootdir,snippets,vztmpl,images,backup
maxfiles 1

 

[Frosty]

It was a firewall issue.

 

[rolandmade]

Thanks for info and hint. It seems to be port 60001 as I saw in the logs.
policy DROP: IN=vmbr5 PHYSIN=enp67s0f3 MAC=ab:cd:00:01:02:c9:b8:96:91:8b:24:ab:08:00 SRC=192.168.39.69 DST=192.168.39.66 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63391 DF PROTO=TCP SPT=57224 DPT=60001 SEQ=2771513959 ACK=0 WINDOW=64240 SYN 105 0 tap105i0-IN 10/Mar/2022:09:04:13 +0100