Can't live migrate after dist-upgrade
- Thread starter churnd
- Start date
Sorry, no -- on the qemu-devel mailing list, we are waiting for feedback from you, QEMU users, about actual migration results with this patch in place. That is, the maintainer with jurisdiction (Gerd Hoffmann) will pick up my patch only *after* users test it and report success. So, please do that -- from your comment above, it looks like you know your way around git; please consider applying the patch directly on your end, and building your own QEMU binary with it. If everything goes well, the patch can be in QEMU 2.6.
You can download the patch (for git-am) from https://patchwork.ozlabs.org/patch/584876/ , clicking the "mbox" link.
Please respond with your results in the following thread:
http://thread.gmane.org/gmane.comp.emulators.qemu/395014
Thanks
Laszlo
Hi, some news.
We have 2 bugs here
The one from qemu 2.5 (qemu 2.5->2.4)
The patch from Iersek has been applied to proxmox package, I have build it for you.
http://odisoweb1.odiso.net/pve-qemu-kvm_2.5-7_amd64.deb
But a bug has been introduce in qemu-server package some months ago,
and -machine flag is not send anymore during the migration
The patch is applied to proxmox git but package not yet release, you can download it here:
http://odisoweb1.odiso.net/qemu-server_4.0-56_amd64.deb
(restart /etc/init.d/pvedaemon after install)
We have 2 bugs here
The one from qemu 2.5 (qemu 2.5->2.4)
The patch from Iersek has been applied to proxmox package, I have build it for you.
http://odisoweb1.odiso.net/pve-qemu-kvm_2.5-7_amd64.deb
But a bug has been introduce in qemu-server package some months ago,
and -machine flag is not send anymore during the migration
The patch is applied to proxmox git but package not yet release, you can download it here:
http://odisoweb1.odiso.net/qemu-server_4.0-56_amd64.deb
(restart /etc/init.d/pvedaemon after install)
Hi,
i am running the most current version 4.1-15 of pve-no-subscription repo.
Today I have installled the newest proxmox packagesve-qemu-kvm_2.5-8_amd64.deb and qemu-server_4.0-62_amd64.deb
I now cannot migrate running vms from kvm 2.4 to 2.5. Even if I restart the vms and they are running kvm 2.5 I cannot migrate them to another kvm 2.5 host and I restarted all pve-daemons.
What packages/patches should I install to be able to migrate again ?
i am running the most current version 4.1-15 of pve-no-subscription repo.
Today I have installled the newest proxmox packages
ve-qemu-kvm_2.5-8_amd64.deb and qemu-server_4.0-62_amd64.debI now cannot migrate running vms from kvm 2.4 to 2.5. Even if I restart the vms and they are running kvm 2.5 I cannot migrate them to another kvm 2.5 host and I restarted all pve-daemons.
What packages/patches should I install to be able to migrate again ?
I'm not getting an error message, my guests simply freeze and become unresponsive...
It seems like the SSH tunnel issue from long ago. I think it's an ssh tunnel being closed before completely finishing the task. However setting migration_unsecure: 1 in datacenter.cfg doesn't seem to disable the tunnel. I'm saying this because when I migrate, the on screen log says it's opening an SSH tunnel.
If I put something like "migration_unsecure 1" without the colon the log says it's ignoring the flag... So it is reading the file...
I'm using the PVE-no-subscription repos as well. Does the enterprise repos have the same problem
It seems like the SSH tunnel issue from long ago. I think it's an ssh tunnel being closed before completely finishing the task. However setting migration_unsecure: 1 in datacenter.cfg doesn't seem to disable the tunnel. I'm saying this because when I migrate, the on screen log says it's opening an SSH tunnel.
If I put something like "migration_unsecure 1" without the colon the log says it's ignoring the flag... So it is reading the file...
I'm using the PVE-no-subscription repos as well. Does the enterprise repos have the same problem
Last edited:
So I tried apt-get update && upgrade on one of my servers, so it looks like this:
proxmox-ve: 4.1-37 (running kernel: 4.2.8-1-pve)
pve-manager: 4.1-13 (running version: 4.1-13/cfb599fb)
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.2.8-1-pve: 4.2.8-37
pve-kernel-4.2.0-1-pve: 4.2.0-13
pve-kernel-4.2.1-1-pve: 4.2.1-14
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-32
qemu-server: 4.0-55
pve-firmware: 1.1-7
libpve-common-perl: 4.0-48
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-40
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-5
pve-container: 1.0-44
pve-firewall: 2.0-17
pve-ha-manager: 1.0-21
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
The old ones in my cluster looks like this:
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.0-1-pve: 4.2.0-13
pve-kernel-4.2.1-1-pve: 4.2.1-14
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-30
qemu-server: 4.0-46
pve-firmware: 1.1-7
libpve-common-perl: 4.0-43
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-21
pve-container: 1.0-37
pve-firewall: 2.0-15
pve-ha-manager: 1.0-18
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
I can't migrate debian VM from the old server to the new one (qemu 2.4.1 -> 2.5.0), it simply fails to start with "Failed to create message: Input/output error". Windows VM migration && startup runs fine.
I created the same debian VM on qemu 2.5.0 (the same configuration), it starts fine, but I am not able to migrate it back to the old hosts in the cluster (qemu 2.4.1)
Did you test it before putting it in the wild?
Any solution?
proxmox-ve: 4.1-37 (running kernel: 4.2.8-1-pve)
pve-manager: 4.1-13 (running version: 4.1-13/cfb599fb)
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.2.8-1-pve: 4.2.8-37
pve-kernel-4.2.0-1-pve: 4.2.0-13
pve-kernel-4.2.1-1-pve: 4.2.1-14
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-32
qemu-server: 4.0-55
pve-firmware: 1.1-7
libpve-common-perl: 4.0-48
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-40
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-5
pve-container: 1.0-44
pve-firewall: 2.0-17
pve-ha-manager: 1.0-21
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
The old ones in my cluster looks like this:
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.0-1-pve: 4.2.0-13
pve-kernel-4.2.1-1-pve: 4.2.1-14
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-30
qemu-server: 4.0-46
pve-firmware: 1.1-7
libpve-common-perl: 4.0-43
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-21
pve-container: 1.0-37
pve-firewall: 2.0-15
pve-ha-manager: 1.0-18
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
I can't migrate debian VM from the old server to the new one (qemu 2.4.1 -> 2.5.0), it simply fails to start with "Failed to create message: Input/output error". Windows VM migration && startup runs fine.
I created the same debian VM on qemu 2.5.0 (the same configuration), it starts fine, but I am not able to migrate it back to the old hosts in the cluster (qemu 2.4.1)
Did you test it before putting it in the wild?
Any solution?
you also need to update qemu-server package on source node.
(you can do dist-upgrade on source node, no impact on running vm)
Hi,
i am running the most current versions on all my servers and migration is not working.
Not from 2.4 to 2.5 and not from 2.5 to 2.5.
I have tried with and without migration_unsecure
What packages should I Install ? The qemu-server 4.0-62 from the repo or the older private patched versions mentioned above ?
i am running the most current versions on all my servers and migration is not working.
Not from 2.4 to 2.5 and not from 2.5 to 2.5.
I have tried with and without migration_unsecure
What packages should I Install ? The qemu-server 4.0-62 from the repo or the older private patched versions mentioned above ?
When qemu-server is updated you need to make a full stop and restart of the VM to use the new qemu version or, if possible, make a live migration.
Hi,
my problem is: I have stopped and started the virtual machine
I have double checked in qemu monitor with info version that my vm is
2.5.0pve-qemu-kvm_2.5-8
The Server I migrate to also has the same kvm version
Still I cannot do live migrations.
I receive no useful debug info, just:
my problem is: I have stopped and started the virtual machine
I have double checked in qemu monitor with info version that my vm is
2.5.0pve-qemu-kvm_2.5-8
The Server I migrate to also has the same kvm version
Still I cannot do live migrations.
I receive no useful debug info, just:
What can I do to be able to live migrate ?
Ok. But normally an upgrade to qemu-server will require a cold reboot to use the upgraded version.
Hi,
my source is: qemu-server: 4.0-62
my dest is: qemu-server: 4.0-62
I had problems with migration also before update. Don't know why, firewall allows everything, net is 10G, Ram is quite high 102 G, Storage is Ceph, Cache is Writeback, mtu is 9000
Maybe something in my setup is special ?
can you post the kvm command line on the source node
"ps -aux|grep "kvm -id <vmid>"
and we you start the migration, try to catch the same command line on target server. (you need to catch it after migration start and before migration stop)
Ok, so it seem to be ok for the command line.
to have more info, you can try to do migration manually.
on the target host, copy/paste the same command line, but remove " -daemonize". (also try to escape -drive '....').
This will launch the target vm in foreground, waiting for migration.
then, on the gui of the sourcevm , in the monitor panel, type
migrate tcp:213.164.159.135:60000.
If the migration hang, you'll see the error in targetvm process.
to have more info, you can try to do migration manually.
on the target host, copy/paste the same command line, but remove " -daemonize". (also try to escape -drive '....').
This will launch the target vm in foreground, waiting for migration.
then, on the gui of the sourcevm , in the monitor panel, type
migrate tcp:213.164.159.135:60000.
If the migration hang, you'll see the error in targetvm process.