Cannot Migrate VMs Twice

[mleo2003]

Hello, I'm having the strangest problem during migration of a 4 node cluster of PVE 5.4 to PVE 6.

I followed the instructions from the wiki and updated Corosync to version 3 successfully. I was able to migrate servers from one node to others, and update it to PVE 6 just fine. Now, I am attempting to migrate some of those nodes back to the new instance (or any other instance), and they fail to migrate. I was able to rerun the kvm command on the target node to see what the issue was, and it shows me this:

Configuration file 'nodes/pve1/qemu-server/153.conf' does not exist

I check, and sure enough, that file doesn't exist there, it still exists in 'nodes/pve2/qemu-server`. I tried to manually copy the config file over as a test, and I get this...

root@pve1:~# cp /etc/pve/nodes/pve2/qemu-server/153.conf /etc/pve/nodes/pve1/qemu-server/
cp: cannot create regular file '/etc/pve/nodes/pve1/qemu-server/153.conf': File exists

This confuses me greatly. I cannot see the file via ls, cat, vim, lsof, or any other method available to me.

Is there something I'm missing now? The cluster still works correctly, I can create new VMs and CTs just fine, etc. I was hoping to finish migrating the hosts to PVE 6 soon, but I really didn't want to have to stop the VMs and leave them on the host while upgrading...

 

[dietmar]

I check, and sure enough, that file doesn't exist there, it still exists in 'nodes/pve2/qemu-server`. I tried to manually copy the config file over as a test, and I get this...

VM config files may only exists once inside a cluster, so at least that cp error is quite obvious.

 

[fabian]

it would also be helpful to include the migration log to see why the migration fails.

 

[mleo2003]

This is what I get when I try the migration from the source host:

root@pve2:~# qm migrate 153 pve1 --online
2019-07-19 08:31:01 starting migration of VM 153 to node 'pve1' (10.10.11.37)
2019-07-19 08:31:02 copying disk images
2019-07-19 08:31:02 starting VM 153 on remote node 'pve1'
2019-07-19 08:31:06 start failed: command '/usr/bin/kvm -id 153 -name foreman-2 -chardev 'socket,id=qmp,path=/var/run/qemu-server/153.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/153.pid -daemonize -smbios 'type=1,uuid=9be130cd-3f21-4cff-8342-bff5018be804' -smp '8,sockets=8,cores=1,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/153.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 12288 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'vmgenid,guid=642c5c20-2434-40a3-a495-d484327ce937' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:1dc7f846b598' -drive 'file=/dev/pve-vg01/vm-153-disk-0,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -netdev 'type=tap,id=net0,ifname=tap153i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=00:50:56:9a:e4:ab,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=pc-i440fx-2.12' -incoming unix:/run/qemu-server/153.migrate -S' failed: exit code 1
2019-07-19 08:31:06 ERROR: online migrate failure - command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve1' root@10.10.11.37 qm start 153 --skiplock --migratedfrom pve2 --migration_type secure --stateuri unix --machine pc-i440fx-2.12' failed: exit code 255
2019-07-19 08:31:06 aborting phase 2 - cleanup resources
2019-07-19 08:31:06 migrate_cancel
2019-07-19 08:31:08 ERROR: migration finished with problems (duration 00:00:07)
migration problems

Given the exit code 1, I tried that command by hand on the target server (pve1) and got the following:

root@pve1:~# qm start 153 --skiplock --migratedfrom pve2 --migration_type secure --stateuri unix --machine pc-i440fx-2.
kvm: error: failed to set MSR 0x10a to 0x0
kvm: /home/build/pve-qemu/pve-qemu-kvm-3.0.1/target/i386/kvm.c:2145: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
start failed: command '/usr/bin/kvm -id 153 -name foreman-2 -chardev 'socket,id=qmp,path=/var/run/qemu-server/153.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/153.pid -daemonize -smbios 'type=1,uuid=9be130cd-3f21-4cff-8342-bff5018be804' -smp '8,sockets=8,cores=1,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/153.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 12288 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'vmgenid,guid=642c5c20-2434-40a3-a495-d484327ce937' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:1dc7f846b598' -drive 'file=/dev/pve-vg01/vm-153-disk-0,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -netdev 'type=tap,id=net0,ifname=tap153i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=00:50:56:9a:e4:ab,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=pc-i440fx-2.12' -incoming unix:/run/qemu-server/153.migrate -S' failed: exit code 1
r

And when I try that last command itself, I get the configuration file not found error, so that makes sense if something else is erroring before there, but the script keeps going anyway.

 

[mleo2003]

A quick google of the kvm errors, and I find these pages:

https://bugzilla.redhat.com/show_bug.cgi?id=1655826
https://access.redhat.com/errata/RHBA-2019:1293

Given this is a patch from Redhat, I don't think I can just apply it within Debian. Should I try downgrading my qemu?

 

[mleo2003]

And this post, from this forum, was ultimately my answer...

https://forum.proxmox.com/threads/vms-not-starting.55489/page-2#post-255783

So, in summary: don't update packages without reboots. It can cause brokenness.