Online migration --with-local-discs failing

Andrey Dudin

New Member
Dec 16, 2017
10
0
1
32
I am trying to run live migration(--with-local-discs --online) in a cluster of proxmoxes with local disks. But it's always failed.

pveversion -v
Code:
proxmox-ve: 5.3-1 (running kernel: 4.15.18-10-pve)
pve-manager: 5.3-9 (running version: 5.3-9/ba817b29)
pve-kernel-4.15: 5.3-1
pve-kernel-4.15.18-10-pve: 4.15.18-32
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-45
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-37
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-34
pve-docs: 5.3-2
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-17
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 3.10.1-1
qemu-server: 5.0-46
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1


lvs -a before start migration
Code:
  LV              VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data            pve     twi-a-tz-- 794.29g             0.00   0.04                        
  [data_tdata]    pve     Twi-ao---- 794.29g                                                
  [data_tmeta]    pve     ewi-ao----   8.11g                                                
  [lvol0_pmspare] pve     ewi-------   8.11g                                                
  root            pve     -wi-ao----  96.00g                                                
  swap            pve     -wi-ao----   8.00g                                                
  data            pve-hdd twi-aotz--   3.45t             3.89   0.52                        
  [data_tdata]    pve-hdd Twi-ao----   3.45t                                                
  [data_tmeta]    pve-hdd ewi-ao----   2.00g                                                
  [lvol0_pmspare] pve-hdd ewi-------   4.00m                                                
  vm-118-disk-1   pve-hdd Vwi-aotz--  10.00g data        95.36                              
  vm-118-disk-2   pve-hdd Vwi-aotz-- 100.00g data        93.95                              
  vm-124-disk-1   pve-hdd Vwi-aotz--  10.00g data        94.92                              
  vm-124-disk-2   pve-hdd Vwi-aotz-- 100.00g data        8.24                                
  vm-139-disk-1   pve-hdd Vwi-aotz--  10.00g data        65.94                              
  vm-139-disk-2   pve-hdd Vwi-aotz-- 100.00g data        9.85                                
  data            pve-ssd twi-a-tz-- 848.11g             0.00   0.34                        
  [data_tdata]    pve-ssd Twi-ao---- 848.11g                                                
  [data_tmeta]    pve-ssd ewi-ao----   1.00g                                                
  [lvol0_pmspare] pve-ssd ewi-------   4.00m

end of migration log:
Code:
2019-02-09 04:41:05 migration status: active (transferred 6753966999, remaining 18206720), total 8607571968)
2019-02-09 04:41:05 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 5865 overflow 0
2019-02-09 04:41:05 migration speed: 7.39 MB/s - downtime 87 ms
2019-02-09 04:41:05 migration status: completed
drive-scsi1: transferred: 109033947136 bytes remaining: 0 bytes total: 109033947136 bytes progression: 100.00 % busy: 0 ready: 1
drive-scsi0: transferred: 10738335744 bytes remaining: 0 bytes total: 10738335744 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
drive-scsi1: Completing block job...
drive-scsi1: Completed successfully.
drive-scsi0: Completing block job...
drive-scsi0: Completed successfully.
drive-scsi1: Cancelling block job
drive-scsi0: Cancelling block job
drive-scsi1: Cancelling block job
drive-scsi0: Cancelling block job
2019-02-09 04:41:12 ERROR: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=prox9' root@10.77.0.6 pvesm free local-hdd:vm-123-disk-1' failed: exit code 5
2019-02-09 04:41:17 ERROR: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=prox9' root@10.77.0.6 pvesm free local-hdd:vm-123-disk-0' failed: exit code 5
2019-02-09 04:41:17 ERROR: Failed to completed storage migration
2019-02-09 04:41:17 ERROR: migration finished with problems (duration 00:18:44)
migration problems

Run failed command manually:
Code:
# /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=prox9' root@10.77.0.6 pvesm free local-hdd:vm-123-disk-1
lvremove 'pve-hdd/vm-123-disk-1' error:   Logical volume pve-hdd/vm-123-disk-1 in use.

lvs -a after migration failed:
Code:
lvs -a | grep 123
  vm-123-disk-0   pve-hdd Vwi-aotz--  10.00g data        100.00                              
  vm-123-disk-1   pve-hdd Vwi-aotz-- 100.00g data        100.00


ps -ef | grep 123 on destination node after migration failed
Code:
root      1757     1 15 04:22 ?        00:03:13 /usr/bin/kvm -id 123 -name vm1 -chardev socket,id=qmp,path=/var/run/qemu-server/123.qmp,server,nowait -mon chardev=qmp,mode=control -pidfile /var/run/qemu-server/123.pid -daemonize -smbios type=1,uuid=29a479d1-b895-4b73-9cdc-c9be185ed4a4 -smp 8,sockets=1,cores=8,maxcpus=8 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/123.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 8192 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/123.qga,server,nowait,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:78d330a2f84 -drive file=/mnt/pve/poseidon/template/iso/ubuntu-16.04.3-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/dev/pve-hdd/vm-123-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -drive file=/dev/pve-hdd/vm-123-disk-1,if=none,id=drive-scsi1,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1 -netdev type=tap,id=net0,ifname=tap123i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=AA:98:18:2C:72:3E,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -machine type=pc-i440fx-2.11 -incoming unix:/run/qemu-server/123.migrate -S
 
Last edited:
You must have same Hardware/Processor type on both nodes, try Default kvm64. Works for me.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!