Migrate Failure to some nodes

[Velas]

I'm getting migration failures when attempting to migrate to some nodes in a 5 node cluster.

pve1 and pv3 - can move VM around, no problem.

attempting to migrate to pve4 or pve5 fails. (pve2 is temporarily offline until I get to data center).

I created a shell of a VM on pve4 just now, and it migrates across all nodes fine. I'm at a loss as to what could be causing this.

task started by HA resource agent
2019-03-28 12:13:13 starting migration of VM 100 to node 'pve4' (192.168.88.104)
2019-03-28 12:13:13 copying disk images
2019-03-28 12:13:13 starting VM 100 on remote node 'pve4'
2019-03-28 12:13:15 start failed: command '/usr/bin/kvm -id 100 -name Ubuntu-Desktop-Test -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=c2452315-ba26-4069-a610-387455c22e18' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga qxl -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4512 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -chardev 'socket,path=/var/run/qemu-server/100.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -spice 'tls-port=61000,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on' -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' -chardev 'spicevmc,id=vdagent,name=vdagent' -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f1ca8915eb50' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=rbd:RBD/vm-100-disk-1:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/RBD_vm.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=rbd:RBD/vm-100-disk-2:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/RBD_vm.keyring,if=none,id=drive-scsi1,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=DA:C23:22:6C:CF,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=pc-i440fx-2.11' -incoming unix:/run/qemu-server/100.migrate -S' failed: exit code 1
2019-03-28 12:13:15 ERROR: online migrate failure - command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve4' root@192.168.88.104 qm start 100 --skiplock --migratedfrom pve3 --migration_type secure --stateuri unix --machine pc-i440fx-2.11' failed: exit code 255
2019-03-28 12:13:15 aborting phase 2 - cleanup resources
2019-03-28 12:13:15 migrate_cancel
2019-03-28 12:13:16 ERROR: migration finished with problems (duration 00:00:03)
TASK ERROR: migration problems

 

[wolfgang]

Hi,

Are these nodes all the same HW?

Try to migrate without HA to get more information.

 

[Velas]

I've identified the root issue. Somehow virtualization support is not enabled on these 2 nodes in BIOS/EFI.

The new VM I mentioned migrating without error was not a good test, because I had quickly created a shell VM with no data, and had not actually tried to start it on the problematic nodes. Attempting to do so confirmed the underlying KVM issue, after which I confirmed KVM is showing virtualization support is not enabled.

A trip to the data center is tentatively scheduled for next Tuesday night, after which I expect migration to work as expected.

Thanks for your input wolfgang.