Online migration --with-local-discs failing

[Andrey Dudin]

I am trying to run live migration(--with-local-discs --online) in a cluster of proxmoxes with local disks. But it's always failed.

pveversion -v

proxmox-ve: 5.3-1 (running kernel: 4.15.18-10-pve)
pve-manager: 5.3-9 (running version: 5.3-9/ba817b29)
pve-kernel-4.15: 5.3-1
pve-kernel-4.15.18-10-pve: 4.15.18-32
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-45
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-37
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-34
pve-docs: 5.3-2
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-17
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 3.10.1-1
qemu-server: 5.0-46
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

lvs -a before start migration

  LV              VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data            pve     twi-a-tz-- 794.29g             0.00   0.04                        
  [data_tdata]    pve     Twi-ao---- 794.29g                                                
  [data_tmeta]    pve     ewi-ao----   8.11g                                                
  [lvol0_pmspare] pve     ewi-------   8.11g                                                
  root            pve     -wi-ao----  96.00g                                                
  swap            pve     -wi-ao----   8.00g                                                
  data            pve-hdd twi-aotz--   3.45t             3.89   0.52                        
  [data_tdata]    pve-hdd Twi-ao----   3.45t                                                
  [data_tmeta]    pve-hdd ewi-ao----   2.00g                                                
  [lvol0_pmspare] pve-hdd ewi-------   4.00m                                                
  vm-118-disk-1   pve-hdd Vwi-aotz--  10.00g data        95.36                              
  vm-118-disk-2   pve-hdd Vwi-aotz-- 100.00g data        93.95                              
  vm-124-disk-1   pve-hdd Vwi-aotz--  10.00g data        94.92                              
  vm-124-disk-2   pve-hdd Vwi-aotz-- 100.00g data        8.24                                
  vm-139-disk-1   pve-hdd Vwi-aotz--  10.00g data        65.94                              
  vm-139-disk-2   pve-hdd Vwi-aotz-- 100.00g data        9.85                                
  data            pve-ssd twi-a-tz-- 848.11g             0.00   0.34                        
  [data_tdata]    pve-ssd Twi-ao---- 848.11g                                                
  [data_tmeta]    pve-ssd ewi-ao----   1.00g                                                
  [lvol0_pmspare] pve-ssd ewi-------   4.00m

end of migration log:

2019-02-09 04:41:05 migration status: active (transferred 6753966999, remaining 18206720), total 8607571968)
2019-02-09 04:41:05 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 5865 overflow 0
2019-02-09 04:41:05 migration speed: 7.39 MB/s - downtime 87 ms
2019-02-09 04:41:05 migration status: completed
drive-scsi1: transferred: 109033947136 bytes remaining: 0 bytes total: 109033947136 bytes progression: 100.00 % busy: 0 ready: 1
drive-scsi0: transferred: 10738335744 bytes remaining: 0 bytes total: 10738335744 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
drive-scsi1: Completing block job...
drive-scsi1: Completed successfully.
drive-scsi0: Completing block job...
drive-scsi0: Completed successfully.
drive-scsi1: Cancelling block job
drive-scsi0: Cancelling block job
drive-scsi1: Cancelling block job
drive-scsi0: Cancelling block job
2019-02-09 04:41:12 ERROR: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=prox9' root@10.77.0.6 pvesm free local-hdd:vm-123-disk-1' failed: exit code 5
2019-02-09 04:41:17 ERROR: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=prox9' root@10.77.0.6 pvesm free local-hdd:vm-123-disk-0' failed: exit code 5
2019-02-09 04:41:17 ERROR: Failed to completed storage migration
2019-02-09 04:41:17 ERROR: migration finished with problems (duration 00:18:44)
migration problems

Run failed command manually:

# /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=prox9' root@10.77.0.6 pvesm free local-hdd:vm-123-disk-1
lvremove 'pve-hdd/vm-123-disk-1' error:   Logical volume pve-hdd/vm-123-disk-1 in use.

lvs -a after migration failed:

lvs -a | grep 123
  vm-123-disk-0   pve-hdd Vwi-aotz--  10.00g data        100.00                              
  vm-123-disk-1   pve-hdd Vwi-aotz-- 100.00g data        100.00

ps -ef | grep 123 on destination node after migration failed

root      1757     1 15 04:22 ?        00:03:13 /usr/bin/kvm -id 123 -name vm1 -chardev socket,id=qmp,path=/var/run/qemu-server/123.qmp,server,nowait -mon chardev=qmp,mode=control -pidfile /var/run/qemu-server/123.pid -daemonize -smbios type=1,uuid=29a479d1-b895-4b73-9cdc-c9be185ed4a4 -smp 8,sockets=1,cores=8,maxcpus=8 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/123.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 8192 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/123.qga,server,nowait,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:78d330a2f84 -drive file=/mnt/pve/poseidon/template/iso/ubuntu-16.04.3-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/dev/pve-hdd/vm-123-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -drive file=/dev/pve-hdd/vm-123-disk-1,if=none,id=drive-scsi1,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1 -netdev type=tap,id=net0,ifname=tap123i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=AA:98:18:2C:72:3E,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -machine type=pc-i440fx-2.11 -incoming unix:/run/qemu-server/123.migrate -S

 

[elmacus]

You must have same Hardware/Processor type on both nodes, try Default kvm64. Works for me.

 

[Andrey Dudin]

You must have same Hardware/Processor type on both nodes, try Default kvm64. Works for me.

I have totally identical hardware ot both servers.

 

[virtRoo]

lvs -a before start migration

Was this output from the source or destination host?

 

[elmacus]

I have totally identical hardware ot both servers.

Did you try to change Processor type in Proxmox ? What setting do you have now in your VM?