Can't live migrate between Proxmox 4.0-16 and 4.0-19

[rkl]

I'm getting a live migration failure between any two Proxmox nodes in a cluster (of 5) where
one node is version 4.0-16 of proxmox-ve and the other node is version 4.0-19. The end of
the pretty unhelpful error mesages is no doubt familiar:

Nov 04 16:54:17 ERROR: online migrate failure - aborting
Nov 04 16:54:17 aborting phase 2 - cleanup resources
Nov 04 16:54:17 migrate_cancel
Nov 04 16:54:18 ERROR: migration finished with problems (duration 00:00:06)
TASK ERROR: migration problems

This tells exactly zero useful info and the various logs on the system don't reveal anything of any help either. It would
be nice to know what the "migration problems" actually are! I'm using iSCSI for storage, so it's not a local storage
issue.

OIder node:
proxmox-ve: 4.0-16 (running kernel: 4.2.2-1-pve)
pve-manager: 4.0-50 (running version: 4.0-50/d3a6b7e5)
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve1
corosync-pve: 2.3.5-1
libqb0: 0.17.2-1
pve-cluster: 4.0-23
qemu-server: 4.0-31
pve-firmware: 1.1-7
libpve-common-perl: 4.0-32
libpve-access-control: 4.0-9
libpve-storage-perl: 4.0-27
pve-libspice-server1: 0.12.5-1
vncterm: 1.2-1
pve-qemu-kvm: 2.4-10
pve-container: 1.0-10
pve-firewall: 2.0-12
pve-ha-manager: 1.0-10
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.3-1
lxcfs: 0.9-pve2
cgmanager: 0.37-pve2
criu: 1.6.0-1
zfsutils: 0.6.5-pve4~jessie

Newer node:
proxmox-ve: 4.0-19 (running kernel: 4.2.3-2-pve)
pve-manager: 4.0-57 (running version: 4.0-57/cc7c2b53)
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-19
lvm2: 2.02.116-pve1
corosync-pve: 2.3.5-1
libqb0: 0.17.2-1
pve-cluster: 4.0-24
qemu-server: 4.0-35
pve-firmware: 1.1-7
libpve-common-perl: 4.0-36
libpve-access-control: 4.0-9
libpve-storage-perl: 4.0-29
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-12
pve-container: 1.0-20
pve-firewall: 2.0-13
pve-ha-manager: 1.0-13
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.4-3
lxcfs: 0.10-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve4~jessie

I tried running the older kernel (4.2.2-1-pve) on the newer node since it was still installed and that didn't fix things.
In desperation, I upgraded the older node to the newer node's version and - bingo - live migration finally worked.
So it looks like one of the recent Proxmox packages breaks live migration (not for the first time!) between it and slightly
older 4.0 nodes and I've now got to do a whole load of offline migrations to get all my nodes up to the 4.0-19 release...grrr!

My suspicion is that very little QA is done between two consecutive releases (whether that's between two minor releases
like this one or between the last of a major version and the first of the next major version after that) to ensure that
live mgration works both backwards and forwards. Live migration is *crucial* for upgrades so that downtime can be
avoided. All the Proxmox upgrade docs tell you to migrate your VMs away before upgrading a cluster node and yet half the
time it seems like live migration between releases is broken and we have to suffer downtime during upgrades :-(

 

[tom]

please post you VM and your storage config:

> qm config VMID

and:

> cat /etc/pve/storage.cfg

 

[rkl]

> qm config VMID

Some items below have been changed to retain anonymity

bootdisk: virtio0
cores: 4
memory: 16384
name: test01
net0: virtio=01:02:03:04:05:06,bridge=vmbr1
net1: virtio=07:08:09:10:11:12,bridge=vmbr2
onboot: 1
ostype: l26
sockets: 1
virtio0: test01:0.0.0.scsi-12345678901234567,size=100G

> cat /etc/pve/storage.cfg

Again, values have been changed below (matching the config above where necessary) - this
is the entry in storage.cfg that matches the config:

iscsi: test01
target iqn.2006-01.com.test01:tsn.123456789012
portal 192.168.1.1
content images

Sorry for the obfuscation, but some of the info is actually client-sensitive. There is absolutely nothing wrong with either the VM config or the storage config - both work fine with live migration between two nodes both running 4.0-16 or between two nodes both running 4.0-19. They definitely don't work between 4.0-16 and 4.0-19 nodes in either direction, which 100% confirms that 4.0-19 isn't compatible with 4.0-16 w.r.t. live migration. I'm doing my final set of offline migrations tonight (already done about 20 VMs with downtime...I'm not a happy bunny).

 

[lweidig]

We also were unable to perform any live migrations between the original 4.0 version and the updates that came out recently. We had to do all offline migrations and get the machines updated. It is working now between like versions for us. Any chance we will see release notes on what might have been fixed / added in the latest patches sent?

 

[spirit]

We also were unable to perform any live migrations between the original 4.0 version and the updates that came out recently. We had to do all offline migrations and get the machines updated. It is working now between like versions for us. Any chance we will see release notes on what might have been fixed / added in the latest patches sent?

Hi, It seem to be related to the switch of virtual nic roms from pxe to efi.

A fix is already available to keep pxe rom when migrated from qemu 2.2.

I think for now it's better to rollback pve-qemu-kvm package to 2.4-10
http://download.proxmox.com/debian/...on/binary-amd64/pve-qemu-kvm_2.4-10_amd64.deb

 

[dietmar]

Any chance we will see release notes on what might have been fixed / added in the latest patches sent?

Each package includes a changelog, and those changelogs are also online, for example:

ftp://download1.proxmox.com/debian/...ion/binary-amd64/qemu-server_4.0-35.changelog

You can even display those changelogs on the GUI (update panel)

And if you want details, you can get the GIT logs from https://git.proxmox.com/

 

[lweidig]

Excellent, thanks! The changelog in the gui ghosts after installation, so be sure to look before applying. Appreciate the quick reply!

 

[tom]

Excellent, thanks! The changelog in the gui ghosts after installation, so be sure to look before applying. Appreciate the quick reply!

you also see the changelog on host via CLI, e.g. for "qemu-server":

> zless /usr/share/doc/qemu-server/changelog.Debian.gz

 

[spirit]

Hi,

I have send a fix on pve-devel mailing list, it'll be available in qemu-server package >= 4.0-36

https://git.proxmox.com/?p=qemu-server.git;a=commit;h=7bac824e19377bbb4816593c5e77dfe4bda8ce62