Live migration between PVE 4.1 and 4.2

[Jimmy]

I searched for this issue and couldn't see it but apologies if this has already been discussed.

I have an existing 4.1 cluster and I have added a new node on which is running 4.2. I get failures when I try to live migrate between these nodes. Does anyone have any experience with this? Should this be possible?

Thanks in advance for any advice.

 

[dcsapak]

what errors do you get?

 

[Jimmy]

This is from 4.2 to 4.1:

Sep 15 11:36:52 starting migration of VM 210 to node 'pve3-dh4' (172.16.23.3)
Sep 15 11:36:52 copying disk images
Sep 15 11:36:52 starting VM 210 on remote node 'pve3-dh4'
Sep 15 11:36:53 start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 210 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 210 -chardev 'socket,id=qmp,path=/var/run/qemu-server/210.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/210.vnc,x509,password -pidfile /var/run/qemu-server/210.pid -daemonize -smbios 'type=1,uuid=66bd38f1-9f0e-40e8-8a86-61eeaee9ae4a' -name JI02-Win2012R2 -smp '8,sockets=2,cores=4,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga std -no-hpet -cpu 'kvm64,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_relaxed,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce' -m 16348 -k en-gb -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4ca8d3abeb75' -drive 'if=none,id=drive-ide0,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=201' -drive 'file=/mnt/pve/pve7-dh4-rz1-1/images/210/vm-210-disk-1.raw,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap210i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=36:64:39:65:37:39,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-i440fx-2.5' -global 'kvm-pit.lost_tick_policy=discard' -incoming tcp:localhost:60000 -S' failed: exit code 1
Sep 15 11:36:53 ERROR: online migrate failure - command '/usr/bin/ssh -o 'BatchMode=yes' root@172.16.23.3 qm start 210 --stateuri tcp --skiplock --migratedfrom pve7-dh4 --machine pc-i440fx-2.5' failed: exit code 255
Sep 15 11:36:53 aborting phase 2 - cleanup resources
Sep 15 11:36:53 migrate_cancel
Sep 15 11:36:54 ERROR: migration finished with problems (duration 00:00:02)
TASK ERROR: migration problems

I tried running:

/usr/bin/ssh -o 'BatchMode=yes' root@172.16.23.3 qm start 210 --stateuri tcp --skiplock --migratedfrom pve7-dh4 --machine pc-i440fx-2.5

manually, however, I get the following:

Running as unit 210.scope.
kvm: -S: Unsupported machine type
Use -machine help to list supported machines!

Running a

kvm -machine help

as suggested shows that the 4.2 machine has "pc-i440fx-2.5 Standard PC (i440FX + PIIX, 1996) (default)" but when running the same command on 4.1 I can not see this option.

I have tried running the SSH command manually with specifying another machine type but still no luck.

Thanks.

 

[dcsapak]

why do you try to live migrate to a 4.1 node? you should always update the nodes to the latest supported version.
generally live migration works *most of the time* between different version (always from low to high), but i believe you ran into a bug we had
this would be fixed with qemu-server > 4.0-59 on your 4.1 machine

 

[Jimmy]

Thanks for your reply. I want to ensure live migration works between nodes before upgrading the remainder of the hosts in the cluster, however, I am happy to only go from lower to higher.

If I do try to do a migration from 4.1 to 4.2 I get the following error:

Sep 15 13:22:28 starting migration of VM 202 to node 'pve7-dh4' (172.16.23.7)
Sep 15 13:22:28 copying disk images
Sep 15 13:22:28 starting VM 202 on remote node 'pve7-dh4'
Sep 15 13:22:29 starting ssh migration tunnel
Sep 15 13:22:30 starting online/live migration on localhost:60000
Sep 15 13:22:30 migrate_set_speed: 8589934592
Sep 15 13:22:30 migrate_set_downtime: 0.1

From there it hangs and the virtual machine becomes totally unresponsive. Only action is to then stop the migration, then stop the VM, run a qm unlock VMID on the host and start it again to get the machine to be usable.

Look forward to hearing back, thanks for your replies so far.

 

[dcsapak]

could you post the vm config?

qm config <vmid>

 

[Jimmy]

Of course, please find below.

root@pve3-dh4:~# qm config 202
bootdisk: virtio0
cores: 4
ide0: none,media=cdrom
ide2: none,media=cdrom
lock: migrate
memory: 16384
name: JI01-Win2012R2
net0: e1000=36:61:37:61:37:38,bridge=vmbr1
numa: 0
ostype: win8
smbios1: uuid=9c75b1a3-d48f-4e2a-9ae1-7acc6810c402
sockets: 2
virtio0: pve3-dh4-rz1-1:202/vm-202-disk-1.raw,size=250G

Thanks again.

 

[DynFi User]

Hello,

I am jumping on this thread because we have the exact same problem.

We have a shared NFS storage and a three node cluster.

Two nodes (production) are running version : 4.2-15/6669ad2c
The last node (where the server is deployed is running latest : 4.2-18/158720b9

I am trying to migrate our KVM from one node (running 4.2-18/158720b9) to one of the two others…
And I have an error :


()
Sep 15 15:23:54 starting migration of VM 110 to node 'proxmaster' (192.168.210.11)
Sep 15 15:23:54 copying disk images
no pools available to import
Sep 15 15:23:54 starting VM 110 on remote node 'proxmaster'
Sep 15 15:23:55 start failed: command '/usr/bin/kvm -id 110 -chardev 'socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/110.pid -daemonize -smbios 'type=1,uuid=90ddbd52-8d3d-4ddc-87d5-34829087ddc1' -name bckaodb16 -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga qxl -vnc unix:/var/run/qemu-server/110.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -k fr -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -spice 'tls-port=61001,addr=localhost,tls-ciphers=DES-CBC3-SHA,seamless-migration=on' -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' -chardev 'spicevmc,id=vdagent,name=vdagent' -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:243ae5a9f582' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=300' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'file=/mnt/pve/Jaws_KVM/images/110/vm-110-disk-1.qcow2,if=none,id=drive-sata0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=6E:A7:57:C7:94:B8,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=200' -netdev 'type=tap,id=net1,ifname=tap110i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=1A:0D:372:3B:19,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=201' -machine 'type=pc-i440fx-2.6' -incoming unix:/run/qemu-server/110.migrate -S' failed: exit code 1
Sep 15 15:23:55 ERROR: online migrate failure - command '/usr/bin/ssh -o 'BatchMode=yes' root@192.168.210.11 qm start 110 --skiplock --migratedfrom proxmini --stateuri unix --machine pc-i440fx-2.6' failed: exit code 255
Sep 15 15:23:55 aborting phase 2 - cleanup resources
Sep 15 15:23:55 migrate_cancel
Sep 15 15:23:56 ERROR: migration finished with problems (duration 00:00:02)
TASK ERROR: migration problems


So far we have tried the following :

• changing driver from VirtIO to E1000 for network
• changing driver from VirtIO disk to SATA


One interesting thing is that I was able to migrate a KVM from one node running 4.2-15/6669ad2c to the node running 4.2-18/158720b9 and then back to the first one running 4.2-15/6669ad2c


Quite stragne and weired ??

 

[dcsapak]

ok i was wrong, live migration is only supported from lower to higher version, never back.
now i guess that your 4.2-15 machine has qemu 2.5 and the 4.2-18 machine has qemu 2.6

so qemu 2.5 -> 2.6 works
but qemu 2.6 -> 2.5 does not work (because of the machine type pc-i440fx-2.6)

but if you migrate from old to new, the machine type stays the same, so as long you do not power off the vm you could migrate back to qemu 2.5,
but beware, with 2.5,2.6 this works, but it can happen that migrating to a lower qemu version does not work at all

so recommendation:
when upgrading a node, upgrade all
and live migrate only between same versions, or from low to high

 

[dcsapak]

Of course, please find below.

root@pve3-dh4:~# qm config 202
bootdisk: virtio0
cores: 4
ide0: none,media=cdrom
ide2: none,media=cdrom
lock: migrate
memory: 16384
name: JI01-Win2012R2
net0: e1000=36:61:37:61:37:38,bridge=vmbr1
numa: 0
ostype: win8
smbios1: uuid=9c75b1a3-d48f-4e2a-9ae1-7acc6810c402
sockets: 2
virtio0: pve3-dh4-rz1-1:202/vm-202-disk-1.raw,size=250G

Thanks again.

nothing special here, but what type of storage is pve3-dh4-rz1-1?
could you post your storage config?

cat /etc/pve/storage.cfg

 

[Jimmy]

Here is the specific storage in question from my storage.cfg

nfs: pve3-dh4-rz1-1
        server 172.30.3.13
        path /mnt/pve/pve3-dh4-rz1-1
        export /pve3-dh4-rz1-1
        options vers=3,intr,hard
        maxfiles 1
        content images

We can live migrate between the existing 4.1 nodes perfectly with this storage. Thanks again.

 

[DynFi User]

Ok - I can confirm this :

We have upgrade a second node to latest stable : 4.2-18/158720b9

And we were able to migrate our nodes !
Definitively a version issue !


Thx !!

 

[Jimmy]

Any other thoughts in regard to my problem live migrating from 4.1 --> 4.2 please?