Live migration between cluster nodes does not work. Proxmox 6.0.7

[Aleksandr Zubarev]

Live migration between cluster nodes does not work. The console and GUI have the same error:

2019-09-12 10:33:44 starting migration of VM 114 to node 'mn-hdap37' (192.168.225.142)
2019-09-12 10:33:44 found local disk 'DATA:vm-114-disk-0' (in current VM config)
2019-09-12 10:33:44 copying disk images
2019-09-12 10:33:44 starting VM 114 on remote node 'mn-hdap37'
2019-09-12 10:33:46 start remote tunnel
2019-09-12 10:33:47 ssh tunnel ver 1
2019-09-12 10:33:47 starting storage migration
2019-09-12 10:33:47 scsi0: start migration to nbd:192.168.225.142:60000:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
drive-scsi0: transferred: 0 bytes remaining: 34359738368 bytes total: 34359738368 bytes progression: 0.00 % busy: 0 ready: 0
drive-scsi0: Cancelling block job
drive-scsi0: Done.
2019-09-12 10:33:48 ERROR: online migrate failure - mirroring error: drive-scsi0: mirroring has been cancelled
2019-09-12 10:33:48 aborting phase 2 - cleanup resources
2019-09-12 10:33:48 migrate_cancel
2019-09-12 10:33:51 ERROR: migration finished with problems (duration 00:00:08)
TASK ERROR: migration problems
in syslog:
VM 114 qmp command failed - VM 114 qmp command 'block-job-cancel' failed - Block job 'drive-scsi0' not found

I couldn't find any more information

VM work with lvm partitions

pvecm status
Quorum information
------------------
Date: Thu Sep 12 12:06:34 2019
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000003
Ring ID: 1/40
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.225.142
0x00000002 1 192.168.225.143
0x00000003 1 192.168.225.144 (local)

version on all three
pveversion
pve-manager/6.0-7/28984024 (running kernel: 5.0.21-1-pve)

 

[mira]

Please post the output of 'pveversion -v', 'cat /etc/pve/storage.cfg' and 'qm config 114'.

 

[Aleksandr Zubarev]

pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-1-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-7
pve-kernel-helper: 6.0-7
pve-kernel-5.0.21-1-pve: 5.0.21-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
libjs-extjs: 6.0.1-10
libknet1: 1.11-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-8
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-64
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1

cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content images,vztmpl,iso,rootdir,snippets
maxfiles 0

lvm: DATA
vgname DATA
content rootdir,images
shared 0

lvm: HWSYS
vgname hwsys
content images,rootdir
shared 0

qm config 114
agent: 1
balloon: 0
bootdisk: scsi0
cores: 2
cpu: Skylake-Client
hotplug: disk,network,usb,memory,cpu
memory: 16384
name: test
net0: virtio=66:F02:8E:17:C3,bridge=vmbr0,queues=4
numa: 1
ostype: l26
scsi0: DATA:vm-114-disk-0,backup=0,cache=none,format=raw,replicate=0,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=0192fe9f-31e6-4a5a-bc15-1a18432718d9
sockets: 2
vmgenid: 5073c1b0-e585-4c6c-b9f2-465b4cb60a7c

more details: this error appears when the io thread flag is enabled