issue on moving Ubuntu Server VMs from one server to another

Stefano Scoppa · Jan 17, 2019

Hi all,

we run proxmox 5.3-5 on a cluster of 5 servers and the QEMU Agent is installed on every VM.

We noticed whenever we move an Ubuntu Server VM from one server to another, the VM would turn totally unresponsive and from the console we would read this:

Kernel Panic.
The only way to restore the VM is to reboot it.

Usually Windows, CentOS or SLES VMs have very little or no problem at all with the migration task, but the Ubuntu Server, no matter which version we're running, is always ending in a Kernel Panic.

How could we solve this?

Stoiko Ivanov · Jan 17, 2019

The screenshot is not attached - please provide it (or a text-dump if possible!)

Stefano Scoppa · Jan 17, 2019

Hi Stoiko,

the screenshot is taken from the Console, so no text I'm sorry.
here the screenshot attached

Stoiko Ivanov · Jan 17, 2019

* Which CPUs do you have in your PVE-nodes?
* What CPU(s) is configured for the Ubuntu Server VM? (please post the VM's config - `qm config $VMID` (anonymized if needed))

Stefano Scoppa · Jan 17, 2019

* Which CPUs do you have in your PVE-nodes?

PVE-node 1) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)
PVE-node 2) 16 x Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2 Sockets)
PVE-node 3) 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)
PVE-node 4) 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)
PVE-node 5) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)

* What CPU(s) is configured for the Ubuntu Server VM? (please post the VM's config - `qm config $VMID` (anonymized if needed))

root@pve02:~# qm config 128
agent: 1
boot: dcn
bootdisk: scsi0
cores: 4
cpu: kvm64
description: builder02
ide2: none,media=cdrom
memory: 16384
name: builder02
net0: virtio=,bridge=vmbr0
numa: 0
ostype: l26
parent: after_100GB_Increase
scsi0: jbod2-pool0-kvm:128/vm-128-disk-1.qcow2,size=210G
scsihw: virtio-scsi-pci
smbios1: uuid=
sockets: 2
root@pve02:~#

Stoiko Ivanov · Jan 17, 2019

* hmm - which versions of pve are you running (`pveversion -v`)?
* which version of ubuntu is the guest, and which kernel is it running?
* Try updating everything to the latest available versions!
You could try the following 2 options to narrow the issue down some more:

* change the guest's cpu-type to host - and migrate between node 1 and node 5 (same cpu)
* try booting some other kernels on the Ubuntu guest and see if the issue persists
* try booting some other kernels (on both nodes) and see if the issue persists

Stefano Scoppa · Jan 17, 2019

* hmm - which versions of pve are you running (`pveversion -v`)?
PVE:

root@pve02:~# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-8-pve: 4.15.18-28
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.16-3-pve: 4.13.16-50
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1
root@pve02:~#

* which version of ubuntu is the guest, and which kernel is it running?
Ubuntu Guest:
itstaff@builder02:~$ uname -a
Linux builder02 4.4.0-141-generic #167-Ubuntu SMP Wed Dec 5 10:40:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
itstaff@builder02:~$

* Try updating everything to the latest available versions!
I can't on this one and on many others, but I'll create a new Ubuntu VM for this tomorrow.
Then, I'll try your tips.

Thanks!

Stefano Scoppa · Jan 18, 2019

Hello, I'm back

so, following your directions I created a new Ubuntu Server VM, latest version available:

Code:

Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-43-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Fri Jan 18 10:41:04 UTC 2019

  System load:  0.27               Processes:            101
  Usage of /:   13.4% of 30.38GB   Users logged in:      0
  Memory usage: 7%                 IP address for ens18:
  Swap usage:   0%

 * MicroK8s is Kubernetes in a snap. Made by devs for devs.
   One quick install on a workstation, VM, or appliance.

   - https://bit.ly/microk8s

 * Full K8s GPU support is now available!

   - https://blog.ubuntu.com/2018/12/10/using-gpgpus-with-kubernetes


 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

3 packages can be updated.
0 updates are security updates.


Last login: Fri Jan 18 10:33:58 2019
itstaff@ubuntu-test-pve:~$ uname -a
Linux ubuntu-test-pve 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
itstaff@ubuntu-test-pve:~$

Originally the config was like this:

Code:

root@pve01:~# qm config 226
agent: 1
bootdisk: scsi0
cores: 1
description: IT-3559%0Au%3A itstaff
ide2: none,media=cdrom
memory: 2048
name: ubuntu-test-pve
net0: virtio=,bridge=vmbr0
numa: 0
ostype: l26
parent: clean
scsi0: jbod2-pool0-kvm:226/vm-226-disk-0.qcow2,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=5e7513cc-3789-48db-850d-88d2c73e80e5
sockets: 2
vmgenid: b93f0794-9aba-4285-9de0-8619de741577
root@pve01:~#

and I changed it to this:

Code:

root@pve01:~# qm config 226
agent: 1
bootdisk: scsi0
cores: 1
cpu: host
description: IT-3559%0Au%3A itstaff
ide2: none,media=cdrom
memory: 2048
name: ubuntu-test-pve
net0: virtio=E6:96:1C:DC:4C:A7,bridge=vmbr0
numa: 0
ostype: l26
parent: clean
scsi0: jbod2-pool0-kvm:226/vm-226-disk-0.qcow2,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=5e7513cc-3789-48db-850d-88d2c73e80e5
sockets: 2
vmgenid: b93f0794-9aba-4285-9de0-8619de741577
root@pve01:~#

Then, I migrated the VM 226 from PVE01 to PVE6 (both running 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets))

Code:

2019-01-18 17:19:29 starting migration of VM 226 to node 'pve06' (10.10.32.6)
2019-01-18 17:19:29 copying disk images
2019-01-18 17:19:29 starting VM 226 on remote node 'pve06'
2019-01-18 17:19:31 start remote tunnel
2019-01-18 17:19:32 ssh tunnel ver 1
2019-01-18 17:19:32 starting online/live migration on unix:/run/qemu-server/226.migrate
2019-01-18 17:19:32 migrate_set_speed: 8589934592
2019-01-18 17:19:32 migrate_set_downtime: 0.1
2019-01-18 17:19:32 set migration_caps
2019-01-18 17:19:32 set cachesize: 268435456
2019-01-18 17:19:32 start migrate command to unix:/run/qemu-server/226.migrate
2019-01-18 17:19:33 migration status: active (transferred 264321329, remaining 326983680), total 2165121024)
2019-01-18 17:19:33 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-18 17:19:34 migration speed: 1024.00 MB/s - downtime 14 ms
2019-01-18 17:19:34 migration status: completed
2019-01-18 17:19:34 ERROR: tunnel replied 'ERR: resume failed - VM 226 not running' to command 'resume 226'
2019-01-18 17:19:38 ERROR: migration finished with problems (duration 00:00:09)
TASK ERROR: migration problems

This is a new error, probably related to the new CPU configuration (host). Usually the task would end just fine, then we would get a Kernel Panic.

Stefano Scoppa · Jan 18, 2019

pve01 and pve06 summary attached.

Stefano Scoppa · Jan 21, 2019

Hi,

here we go with a fresh new KP.

from
PVE05 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)
to
PVE06) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)

root@pve05:~# qm config 216
bootdisk: scsi0
cores: 2
description:
ide2: none,media=cdrom
memory: 6144
name: drulli01.it.esteco.net
net0: virtio=1E:7F

6:4B:37:A1,bridge=vmbr0
numa: 0
ostype: l26
scsi0: jbod2-pool0-kvm:216/vm-216-disk-0.qcow2,size=20G
scsihw: virtio-scsi-pci
smbios1: uuid=00cf5840-91a2-48f3-90cd-d2962b285e9c
sockets: 1
vmgenid: 80a77f11-8f01-4f09-bdd3-180ad6d89ee1
root@pve05:~#

itstaff@drulli01:~$ uname -a
Linux drulli01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
itstaff@drulli01:~$

Code:

()

2019-01-21 13:50:23 starting migration of VM 216 to node 'pve06' (10.10.32.6)
2019-01-21 13:50:24 copying disk images
2019-01-21 13:50:24 starting VM 216 on remote node 'pve06'
2019-01-21 13:50:26 start remote tunnel
2019-01-21 13:50:26 ssh tunnel ver 1
2019-01-21 13:50:26 starting online/live migration on unix:/run/qemu-server/216.migrate
2019-01-21 13:50:26 migrate_set_speed: 8589934592
2019-01-21 13:50:26 migrate_set_downtime: 0.1
2019-01-21 13:50:26 set migration_caps
2019-01-21 13:50:26 set cachesize: 1073741824
2019-01-21 13:50:26 start migrate command to unix:/run/qemu-server/216.migrate
2019-01-21 13:50:27 migration status: active (transferred 84359959, remaining 4880265216), total 6460088320)
2019-01-21 13:50:27 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:29 migration status: active (transferred 118625972, remaining 2034929664), total 6460088320)
2019-01-21 13:50:29 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:30 migration status: active (transferred 254848872, remaining 1131745280), total 6460088320)
2019-01-21 13:50:30 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:31 migration status: active (transferred 419685077, remaining 962138112), total 6460088320)
2019-01-21 13:50:31 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:32 migration status: active (transferred 594444457, remaining 784633856), total 6460088320)
2019-01-21 13:50:32 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:33 migration status: active (transferred 764792226, remaining 613306368), total 6460088320)
2019-01-21 13:50:33 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:34 migration status: active (transferred 947500666, remaining 429907968), total 6460088320)
2019-01-21 13:50:34 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:35 migration status: active (transferred 1129693380, remaining 235212800), total 6460088320)
2019-01-21 13:50:35 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:36 migration status: active (transferred 1307168780, remaining 53506048), total 6460088320)
2019-01-21 13:50:36 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:36 migration status: active (transferred 1325694252, remaining 35016704), total 6460088320)
2019-01-21 13:50:36 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:36 migration status: active (transferred 1343392142, remaining 4997120), total 6460088320)
2019-01-21 13:50:36 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 13:50:36 migration speed: 614.40 MB/s - downtime 47 ms
2019-01-21 13:50:36 migration status: completed
2019-01-21 13:50:39 migration finished successfully (duration 00:00:16)
TASK OK

ending in Kernel Panic.
Screenshot attached.

Stefano Scoppa · Jan 21, 2019

another migration, another KP.

from
PVE05 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)
to
PVE06) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)

root@pve05:~# qm config 189
agent: 1
bootdisk: scsi0
cores: 2
description:
ide2: none,media=cdrom
memory: 4096
name: retro.it.esteco.net
net0: virtio=02:E4:75:46:9E:AC,bridge=vmbr0
numa: 0
ostype: l26
scsi0: jbod2-pool0-kvm:189/vm-189-disk-0.qcow2,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=37882f91-0a07-4816-8bcf-ae72d1473082
sockets: 2
vmgenid: e25b6c83-9cba-4316-9939-4573d37475a9
root@pve05:~#

itstaff@retro:~$ uname -a
Linux retro 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
itstaff@retro:~$

Code:

Virtual Environment 5.3-5

Search
Virtual Machine 189 (retro.it.esteco.net) on node 'pve06'


Server View
Logs
()
2019-01-21 16:15:15 starting migration of VM 189 to node 'pve06' (10.10.32.6)
2019-01-21 16:15:15 copying disk images
2019-01-21 16:15:16 starting VM 189 on remote node 'pve06'
2019-01-21 16:15:18 start remote tunnel
2019-01-21 16:15:19 ssh tunnel ver 1
2019-01-21 16:15:19 starting online/live migration on unix:/run/qemu-server/189.migrate
2019-01-21 16:15:19 migrate_set_speed: 8589934592
2019-01-21 16:15:19 migrate_set_downtime: 0.1
2019-01-21 16:15:19 set migration_caps
2019-01-21 16:15:19 set cachesize: 536870912
2019-01-21 16:15:19 start migrate command to unix:/run/qemu-server/189.migrate
2019-01-21 16:15:20 migration status: active (transferred 175581501, remaining 4129947648), total 4312604672)
2019-01-21 16:15:20 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:21 migration status: active (transferred 364041616, remaining 3941777408), total 4312604672)
2019-01-21 16:15:21 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:22 migration status: active (transferred 542645364, remaining 3762786304), total 4312604672)
2019-01-21 16:15:22 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:23 migration status: active (transferred 721957574, remaining 3581919232), total 4312604672)
2019-01-21 16:15:23 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:24 migration status: active (transferred 887748777, remaining 3342970880), total 4312604672)
2019-01-21 16:15:24 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:25 migration status: active (transferred 1158742970, remaining 2812174336), total 4312604672)
2019-01-21 16:15:25 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:26 migration status: active (transferred 1502400508, remaining 2448338944), total 4312604672)
2019-01-21 16:15:26 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:27 migration status: active (transferred 1847657624, remaining 2100154368), total 4312604672)
2019-01-21 16:15:27 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:28 migration status: active (transferred 2199068895, remaining 1748516864), total 4312604672)
2019-01-21 16:15:28 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:29 migration status: active (transferred 2552494385, remaining 1387794432), total 4312604672)
2019-01-21 16:15:29 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:30 migration status: active (transferred 2902502889, remaining 966373376), total 4312604672)
2019-01-21 16:15:30 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:31 migration status: active (transferred 3232477421, remaining 633102336), total 4312604672)
2019-01-21 16:15:31 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:32 migration status: active (transferred 3423584697, remaining 438525952), total 4312604672)
2019-01-21 16:15:32 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:33 migration status: active (transferred 3605659943, remaining 252563456), total 4312604672)
2019-01-21 16:15:33 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:34 migration status: active (transferred 3782724367, remaining 69664768), total 4312604672)
2019-01-21 16:15:34 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:34 migration status: active (transferred 3801512495, remaining 50913280), total 4312604672)
2019-01-21 16:15:34 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:34 migration status: active (transferred 3819249999, remaining 33210368), total 4312604672)
2019-01-21 16:15:34 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:34 migration status: active (transferred 3835084790, remaining 14622720), total 4312604672)
2019-01-21 16:15:34 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-21 16:15:34 migration speed: 273.07 MB/s - downtime 108 ms
2019-01-21 16:15:34 migration status: completed
2019-01-21 16:15:37 migration finished successfully (duration 00:00:22)
TASK OK

ending in Kernel Panic.
Screenshot attached.

root@pve05:~# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-8-pve: 4.15.18-28
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.16-3-pve: 4.13.16-50
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1
root@pve05:~#

root@pve06:~# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-8-pve: 4.15.18-28
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.16-3-pve: 4.13.16-50
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1
root@pve06:~#

Stefano Scoppa · Jan 22, 2019

another migration, another KP.

from
PVE06) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)
to
PVE05 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)

root@pve05:~# qm config 149
agent: 1
bootdisk: scsi0
cores: 2
description:
ide2: none,media=cdrom
memory: 4096
name: keycloak.it.esteco.net
net0: virtio=8E:B2:55:9E:6D:55,bridge=vmbr0
numa: 0
ostype: l26
scsi0: jbod2-pool0-kvm:149/vm-149-disk-0.qcow2,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=11219ddf-8821-4017-ba77-6337e6e03a13
sockets: 2
vmgenid: 7d2e04c6-fb8c-4e91-a39d-cac81602c025
root@pve05:~#

itstaff@keycloak:~$ uname -a
Linux keycloak4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
itstaff@keycloak:~$

Code:

2019-01-22 11:12:13 starting migration of VM 149 to node 'pve05' (10.10.32.5)
2019-01-22 11:12:13 copying disk images
2019-01-22 11:12:13 starting VM 149 on remote node 'pve05'
2019-01-22 11:12:17 start remote tunnel
2019-01-22 11:12:20 ssh tunnel ver 1
2019-01-22 11:12:20 starting online/live migration on unix:/run/qemu-server/149.migrate
2019-01-22 11:12:20 migrate_set_speed: 8589934592
2019-01-22 11:12:20 migrate_set_downtime: 0.1
2019-01-22 11:12:20 set migration_caps
2019-01-22 11:12:20 set cachesize: 536870912
2019-01-22 11:12:20 start migrate command to unix:/run/qemu-server/149.migrate
2019-01-22 11:12:21 migration status: active (transferred 174900867, remaining 4130340864), total 4312604672)
2019-01-22 11:12:21 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:22 migration status: active (transferred 358149100, remaining 3947282432), total 4312604672)
2019-01-22 11:12:22 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:23 migration status: active (transferred 542494929, remaining 3760431104), total 4312604672)
2019-01-22 11:12:23 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:24 migration status: active (transferred 727460939, remaining 3572744192), total 4312604672)
2019-01-22 11:12:24 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:25 migration status: active (transferred 911757574, remaining 3385917440), total 4312604672)
2019-01-22 11:12:25 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:26 migration status: active (transferred 1107985868, remaining 3186581504), total 4312604672)
2019-01-22 11:12:26 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:27 migration status: active (transferred 1271499271, remaining 3017031680), total 4312604672)
2019-01-22 11:12:27 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:28 migration status: active (transferred 1444636536, remaining 2837884928), total 4312604672)
2019-01-22 11:12:28 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:29 migration status: active (transferred 1618764646, remaining 2662535168), total 4312604672)
2019-01-22 11:12:29 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:30 migration status: active (transferred 1785313218, remaining 2494496768), total 4312604672)
2019-01-22 11:12:30 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:31 migration status: active (transferred 1952256548, remaining 2327576576), total 4312604672)
2019-01-22 11:12:31 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:32 migration status: active (transferred 2123898481, remaining 2156183552), total 4312604672)
2019-01-22 11:12:32 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:33 migration status: active (transferred 2296636091, remaining 1983741952), total 4312604672)
2019-01-22 11:12:33 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:34 migration status: active (transferred 2468265406, remaining 1810636800), total 4312604672)
2019-01-22 11:12:34 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:35 migration status: active (transferred 2641000011, remaining 1637695488), total 4312604672)
2019-01-22 11:12:35 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:36 migration status: active (transferred 2815518901, remaining 1463410688), total 4312604672)
2019-01-22 11:12:36 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:37 migration status: active (transferred 2984278575, remaining 1293598720), total 4312604672)
2019-01-22 11:12:37 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:38 migration status: active (transferred 3143850365, remaining 1067286528), total 4312604672)
2019-01-22 11:12:38 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:39 migration status: active (transferred 3311498943, remaining 899956736), total 4312604672)
2019-01-22 11:12:39 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:40 migration status: active (transferred 3622262769, remaining 587665408), total 4312604672)
2019-01-22 11:12:40 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:41 migration status: active (transferred 3854435938, remaining 351526912), total 4312604672)
2019-01-22 11:12:41 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:42 migration status: active (transferred 4022961170, remaining 176594944), total 4312604672)
2019-01-22 11:12:42 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 11:12:43 migration speed: 178.09 MB/s - downtime 11 ms
2019-01-22 11:12:43 migration status: completed
2019-01-22 11:12:46 migration finished successfully (duration 00:00:33)
TASK OK

ending in Kernel Panic.
Screenshot attached.

Stefano Scoppa · Jan 22, 2019

another migration, another KP.

from
PVE06) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)
to
PVE04) 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)

root@pve06:~# qm config 194
agent: 1
boot: dcn
bootdisk: scsi0
cores: 2
cpu: kvm64
description:
ide2: none,media=cdrom
memory: 6144
name: sonar.it.esteco.net
net0: virtio=26:A1:59:3A:4A:6A,bridge=vmbr0
numa: 0
ostype: l26
scsi0: jbod2-pool0-kvm:194/vm-194-disk-1.qcow2,size=40G
scsihw: virtio-scsi-pci
smbios1: uuid=8f818719-f018-44e6-8ae6-3c779c94fad6
sockets: 1
root@pve06:~#

itstaff@sonar:~$ uname -a
Linux sonar 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
itstaff@sonar:~$

Code:

2019-01-22 16:13:10 starting migration of VM 194 to node 'pve04' (10.10.32.4)
2019-01-22 16:13:10 copying disk images
2019-01-22 16:13:10 starting VM 194 on remote node 'pve04'
2019-01-22 16:13:13 start remote tunnel
2019-01-22 16:13:14 ssh tunnel ver 1
2019-01-22 16:13:14 starting online/live migration on unix:/run/qemu-server/194.migrate
2019-01-22 16:13:14 migrate_set_speed: 8589934592
2019-01-22 16:13:14 migrate_set_downtime: 0.1
2019-01-22 16:13:14 set migration_caps
2019-01-22 16:13:14 set cachesize: 1073741824
2019-01-22 16:13:14 start migrate command to unix:/run/qemu-server/194.migrate
2019-01-22 16:13:15 migration status: active (transferred 169269945, remaining 6264913920), total 6460088320)
2019-01-22 16:13:15 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:16 migration status: active (transferred 342871314, remaining 6079549440), total 6460088320)
2019-01-22 16:13:16 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:17 migration status: active (transferred 530421159, remaining 5869555712), total 6460088320)
2019-01-22 16:13:17 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:18 migration status: active (transferred 716075390, remaining 5672431616), total 6460088320)
2019-01-22 16:13:18 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:19 migration status: active (transferred 903671677, remaining 5480431616), total 6460088320)
2019-01-22 16:13:19 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:20 migration status: active (transferred 1077691922, remaining 5301989376), total 6460088320)
2019-01-22 16:13:20 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:21 migration status: active (transferred 1262361211, remaining 5111431168), total 6460088320)
2019-01-22 16:13:21 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:22 migration status: active (transferred 1444706349, remaining 4918185984), total 6460088320)
2019-01-22 16:13:22 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:23 migration status: active (transferred 1624041671, remaining 4732391424), total 6460088320)
2019-01-22 16:13:23 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:24 migration status: active (transferred 1817526666, remaining 4527304704), total 6460088320)
2019-01-22 16:13:24 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:25 migration status: active (transferred 1978916996, remaining 4356739072), total 6460088320)
2019-01-22 16:13:25 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:26 migration status: active (transferred 2140374349, remaining 4189216768), total 6460088320)
2019-01-22 16:13:26 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:27 migration status: active (transferred 2292653672, remaining 4033396736), total 6460088320)
2019-01-22 16:13:27 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:28 migration status: active (transferred 2456113183, remaining 3864145920), total 6460088320)
2019-01-22 16:13:28 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:29 migration status: active (transferred 2623363251, remaining 3680628736), total 6460088320)
2019-01-22 16:13:29 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:30 migration status: active (transferred 2914423006, remaining 3373449216), total 6460088320)
2019-01-22 16:13:30 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:31 migration status: active (transferred 3192032559, remaining 3011366912), total 6460088320)
2019-01-22 16:13:31 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:32 migration status: active (transferred 3353256020, remaining 2831015936), total 6460088320)
2019-01-22 16:13:32 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:33 migration status: active (transferred 3597749245, remaining 2565873664), total 6460088320)
2019-01-22 16:13:33 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:34 migration status: active (transferred 3843328526, remaining 2309652480), total 6460088320)
2019-01-22 16:13:34 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:35 migration status: active (transferred 3993280819, remaining 2156179456), total 6460088320)
2019-01-22 16:13:35 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:36 migration status: active (transferred 4147243926, remaining 1996292096), total 6460088320)
2019-01-22 16:13:36 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:37 migration status: active (transferred 4297011736, remaining 1837326336), total 6460088320)
2019-01-22 16:13:37 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:38 migration status: active (transferred 4451663370, remaining 1677180928), total 6460088320)
2019-01-22 16:13:38 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:39 migration status: active (transferred 4614216221, remaining 1504960512), total 6460088320)
2019-01-22 16:13:39 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:40 migration status: active (transferred 4768695532, remaining 1339375616), total 6460088320)
2019-01-22 16:13:40 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:41 migration status: active (transferred 4914337967, remaining 1186779136), total 6460088320)
2019-01-22 16:13:41 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:42 migration status: active (transferred 5060931865, remaining 1035395072), total 6460088320)
2019-01-22 16:13:42 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:43 migration status: active (transferred 5215291413, remaining 870268928), total 6460088320)
2019-01-22 16:13:43 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:44 migration status: active (transferred 5372606255, remaining 707596288), total 6460088320)
2019-01-22 16:13:44 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:45 migration status: active (transferred 5516490829, remaining 548044800), total 6460088320)
2019-01-22 16:13:45 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:46 migration status: active (transferred 5673386711, remaining 382226432), total 6460088320)
2019-01-22 16:13:46 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:47 migration status: active (transferred 5942661193, remaining 97349632), total 6460088320)
2019-01-22 16:13:47 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:48 migration status: active (transferred 5963944553, remaining 76107776), total 6460088320)
2019-01-22 16:13:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:48 migration status: active (transferred 5984045961, remaining 56045568), total 6460088320)
2019-01-22 16:13:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:48 migration status: active (transferred 6005066665, remaining 35065856), total 6460088320)
2019-01-22 16:13:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:13:48 migration status: active (transferred 6024625312, remaining 66351104), total 6460088320)
2019-01-22 16:13:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 168 overflow 0
2019-01-22 16:13:48 migration status: active (transferred 6043159136, remaining 47788032), total 6460088320)
2019-01-22 16:13:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 4684 overflow 0
2019-01-22 16:13:48 migration status: active (transferred 6058577988, remaining 32350208), total 6460088320)
2019-01-22 16:13:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 8441 overflow 0
2019-01-22 16:13:48 migration status: active (transferred 6061495940, remaining 29437952), total 6460088320)
2019-01-22 16:13:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 9152 overflow 0
2019-01-22 16:13:48 migration status: active (transferred 6071140401, remaining 19791872), total 6460088320)
2019-01-22 16:13:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 11502 overflow 0
2019-01-22 16:13:48 migration speed: 180.71 MB/s - downtime 94 ms
2019-01-22 16:13:48 migration status: completed
2019-01-22 16:13:53 migration finished successfully (duration 00:00:43)
TASK OK

root@pve04:~# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-7 (running version: 5.3-7/e8ed1e22)
pve-kernel-4.15: 5.2-12
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-8-pve: 4.15.18-28
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.16-3-pve: 4.13.16-50
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-19
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-35
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-33
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-44
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1
root@pve04:~#

ending in Kernel Panic.
Screenshot attached.

Stefano Scoppa · Jan 22, 2019

Update: the KP appears also on Ubuntu Desktop.
After a migration from one node to another, the GUI from the Console becomes unresponsive.

Stefano Scoppa · Jan 22, 2019

another migration, another KP.

from
PVE06) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)
to
PVE05 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)

root@pve06:~# qm config 197
agent: 1
bootdisk: virtio0
cores: 4
description:
ide2: none,media=cdrom
memory: 8192
name: devEdi.it.esteco.net
net0: virtio=4E:67:E8:E1:6F:88,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=bbc59afa-2ef7-4885-a3f1-2afbc6a19416
sockets: 1
virtio0: jbod2-pool0-kvm:197/vm-197-disk-1.qcow2,size=40G
root@pve06:~#

itstaff@devEdi:~$ uname -a
Linux devEdi 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
itstaff@devEdi:~$

Code:

2019-01-22 16:39:49 starting migration of VM 197 to node 'pve05' (10.10.32.5)
2019-01-22 16:39:49 copying disk images
2019-01-22 16:39:49 starting VM 197 on remote node 'pve05'
2019-01-22 16:39:51 start remote tunnel
2019-01-22 16:39:53 ssh tunnel ver 1
2019-01-22 16:39:53 starting online/live migration on unix:/run/qemu-server/197.migrate
2019-01-22 16:39:53 migrate_set_speed: 8589934592
2019-01-22 16:39:53 migrate_set_downtime: 0.1
2019-01-22 16:39:53 set migration_caps
2019-01-22 16:39:53 set cachesize: 1073741824
2019-01-22 16:39:53 start migrate command to unix:/run/qemu-server/197.migrate
2019-01-22 16:39:54 migration status: active (transferred 179504835, remaining 8417312768), total 8607571968)
2019-01-22 16:39:54 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:39:55 migration status: active (transferred 331309985, remaining 8264830976), total 8607571968)
2019-01-22 16:39:55 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:39:56 migration status: active (transferred 492427156, remaining 8101171200), total 8607571968)
2019-01-22 16:39:56 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:39:57 migration status: active (transferred 656900813, remaining 7930703872), total 8607571968)
2019-01-22 16:39:57 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:39:58 migration status: active (transferred 824514671, remaining 7756804096), total 8607571968)
2019-01-22 16:39:58 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:39:59 migration status: active (transferred 993925629, remaining 7581323264), total 8607571968)
2019-01-22 16:39:59 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:00 migration status: active (transferred 1160357614, remaining 7412301824), total 8607571968)
2019-01-22 16:40:00 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:01 migration status: active (transferred 1328262243, remaining 7240257536), total 8607571968)
2019-01-22 16:40:01 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:02 migration status: active (transferred 1496560010, remaining 7066341376), total 8607571968)
2019-01-22 16:40:02 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:03 migration status: active (transferred 1664907466, remaining 6892171264), total 8607571968)
2019-01-22 16:40:03 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:04 migration status: active (transferred 1830449900, remaining 6725443584), total 8607571968)
2019-01-22 16:40:04 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:05 migration status: active (transferred 2004011398, remaining 6547046400), total 8607571968)
2019-01-22 16:40:05 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:06 migration status: active (transferred 2162429127, remaining 6380040192), total 8607571968)
2019-01-22 16:40:06 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:07 migration status: active (transferred 2327828047, remaining 6204076032), total 8607571968)
2019-01-22 16:40:07 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:08 migration status: active (transferred 2492979036, remaining 6032850944), total 8607571968)
2019-01-22 16:40:08 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:09 migration status: active (transferred 2661916532, remaining 5854793728), total 8607571968)
2019-01-22 16:40:09 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:10 migration status: active (transferred 2829193447, remaining 5685149696), total 8607571968)
2019-01-22 16:40:10 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:11 migration status: active (transferred 2996553710, remaining 5486895104), total 8607571968)
2019-01-22 16:40:11 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:12 migration status: active (transferred 3159285751, remaining 5255487488), total 8607571968)
2019-01-22 16:40:12 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:13 migration status: active (transferred 3337852707, remaining 5076467712), total 8607571968)
2019-01-22 16:40:13 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:14 migration status: active (transferred 3607445212, remaining 4801544192), total 8607571968)
2019-01-22 16:40:14 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:15 migration status: active (transferred 3768101169, remaining 4635328512), total 8607571968)
2019-01-22 16:40:15 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:16 migration status: active (transferred 3927920099, remaining 4469862400), total 8607571968)
2019-01-22 16:40:16 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:17 migration status: active (transferred 4097527545, remaining 4296278016), total 8607571968)
2019-01-22 16:40:17 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:18 migration status: active (transferred 4264631903, remaining 4125028352), total 8607571968)
2019-01-22 16:40:18 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:19 migration status: active (transferred 4431747862, remaining 3952226304), total 8607571968)
2019-01-22 16:40:19 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:20 migration status: active (transferred 4593067956, remaining 3783806976), total 8607571968)
2019-01-22 16:40:20 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:21 migration status: active (transferred 4759484173, remaining 3616370688), total 8607571968)
2019-01-22 16:40:21 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:22 migration status: active (transferred 4914074472, remaining 3458048000), total 8607571968)
2019-01-22 16:40:22 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:23 migration status: active (transferred 5085570326, remaining 3280453632), total 8607571968)
2019-01-22 16:40:23 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:24 migration status: active (transferred 5241311914, remaining 3119996928), total 8607571968)
2019-01-22 16:40:24 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:25 migration status: active (transferred 5405277512, remaining 2951520256), total 8607571968)
2019-01-22 16:40:25 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:26 migration status: active (transferred 5559839020, remaining 2793254912), total 8607571968)
2019-01-22 16:40:26 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:27 migration status: active (transferred 5712337980, remaining 2634383360), total 8607571968)
2019-01-22 16:40:27 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:28 migration status: active (transferred 5956047313, remaining 2383085568), total 8607571968)
2019-01-22 16:40:28 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:29 migration status: active (transferred 6253345104, remaining 2075287552), total 8607571968)
2019-01-22 16:40:29 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:30 migration status: active (transferred 6469345859, remaining 1847173120), total 8607571968)
2019-01-22 16:40:30 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:31 migration status: active (transferred 6630049515, remaining 1677889536), total 8607571968)
2019-01-22 16:40:31 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:32 migration status: active (transferred 6787388576, remaining 1515376640), total 8607571968)
2019-01-22 16:40:32 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:33 migration status: active (transferred 7027969402, remaining 1264041984), total 8607571968)
2019-01-22 16:40:33 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:34 migration status: active (transferred 7199296740, remaining 1084866560), total 8607571968)
2019-01-22 16:40:34 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:35 migration status: active (transferred 7494197362, remaining 781524992), total 8607571968)
2019-01-22 16:40:35 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:36 migration status: active (transferred 7711190007, remaining 564183040), total 8607571968)
2019-01-22 16:40:36 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:37 migration status: active (transferred 7868655364, remaining 368422912), total 8607571968)
2019-01-22 16:40:37 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:38 migration status: active (transferred 8026525137, remaining 201064448), total 8607571968)
2019-01-22 16:40:38 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:39 migration status: active (transferred 8181649065, remaining 42156032), total 8607571968)
2019-01-22 16:40:39 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:40:39 migration status: active (transferred 8212546569, remaining 78757888), total 8607571968)
2019-01-22 16:40:39 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 1141 overflow 0
2019-01-22 16:40:39 migration status: active (transferred 8235406270, remaining 55758848), total 8607571968)
2019-01-22 16:40:39 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 6711 overflow 0
2019-01-22 16:40:39 migration status: active (transferred 8246015206, remaining 45137920), total 8607571968)
2019-01-22 16:40:39 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 9296 overflow 0
2019-01-22 16:40:39 migration status: active (transferred 8267224820, remaining 23912448), total 8607571968)
2019-01-22 16:40:39 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 14464 overflow 0
2019-01-22 16:40:39 migration speed: 178.09 MB/s - downtime 58 ms
2019-01-22 16:40:39 migration status: completed
2019-01-22 16:40:43 migration finished successfully (duration 00:00:55)
TASK OK

ending in Kernel Panic.
Screenshot attached.

Stefano Scoppa · Jan 22, 2019

another migration, another KP.

from
PVE06) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)
to
PVE05) 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)

root@pve06:~# qm config 200
agent: 1
bootdisk: virtio0
cores: 4
description: Clone of 197 - devEdi
ide2: none,media=cdrom
memory: 8192
name: devEnt.it.esteco.net
net0: virtio=42:A7:5E:34

3:8D,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=c89121a3-a69d-4b73-9229-01b0cd5494a1
sockets: 1
virtio0: jbod2-pool0-kvm:200/vm-200-disk-1.qcow2,size=40G
root@pve06:~#

itstaff@devEnt:~$ uname -a
Linux devEnt 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
itstaff@devEnt:~$

Code:

2019-01-22 16:48:09 starting migration of VM 200 to node 'pve05' (10.10.32.5)
2019-01-22 16:48:09 copying disk images
2019-01-22 16:48:09 starting VM 200 on remote node 'pve05'
2019-01-22 16:48:12 start remote tunnel
2019-01-22 16:48:13 ssh tunnel ver 1
2019-01-22 16:48:13 starting online/live migration on unix:/run/qemu-server/200.migrate
2019-01-22 16:48:13 migrate_set_speed: 8589934592
2019-01-22 16:48:13 migrate_set_downtime: 0.1
2019-01-22 16:48:13 set migration_caps
2019-01-22 16:48:13 set cachesize: 1073741824
2019-01-22 16:48:13 start migrate command to unix:/run/qemu-server/200.migrate
2019-01-22 16:48:14 migration status: active (transferred 311937639, remaining 8282898432), total 8607571968)
2019-01-22 16:48:14 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:15 migration status: active (transferred 655849264, remaining 7935762432), total 8607571968)
2019-01-22 16:48:15 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:16 migration status: active (transferred 988730976, remaining 7598886912), total 8607571968)
2019-01-22 16:48:16 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:17 migration status: active (transferred 1327470589, remaining 7257354240), total 8607571968)
2019-01-22 16:48:17 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:18 migration status: active (transferred 1649779128, remaining 6933471232), total 8607571968)
2019-01-22 16:48:18 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:19 migration status: active (transferred 1853783458, remaining 6726848512), total 8607571968)
2019-01-22 16:48:19 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:20 migration status: active (transferred 2008775643, remaining 6570127360), total 8607571968)
2019-01-22 16:48:20 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:21 migration status: active (transferred 2164411193, remaining 6411337728), total 8607571968)
2019-01-22 16:48:21 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:22 migration status: active (transferred 2346271698, remaining 6226198528), total 8607571968)
2019-01-22 16:48:22 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:23 migration status: active (transferred 2581749452, remaining 5988229120), total 8607571968)
2019-01-22 16:48:23 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:24 migration status: active (transferred 2741117032, remaining 5828763648), total 8607571968)
2019-01-22 16:48:24 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:25 migration status: active (transferred 2893535280, remaining 5674942464), total 8607571968)
2019-01-22 16:48:25 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:26 migration status: active (transferred 3075940852, remaining 5461651456), total 8607571968)
2019-01-22 16:48:26 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:27 migration status: active (transferred 3338285892, remaining 5130502144), total 8607571968)
2019-01-22 16:48:27 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:28 migration status: active (transferred 3560070110, remaining 4905508864), total 8607571968)
2019-01-22 16:48:28 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:29 migration status: active (transferred 3806762529, remaining 4655202304), total 8607571968)
2019-01-22 16:48:29 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:30 migration status: active (transferred 4011931001, remaining 4448047104), total 8607571968)
2019-01-22 16:48:30 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:31 migration status: active (transferred 4179346560, remaining 4278673408), total 8607571968)
2019-01-22 16:48:31 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:32 migration status: active (transferred 4460325335, remaining 3995303936), total 8607571968)
2019-01-22 16:48:32 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:33 migration status: active (transferred 4707981340, remaining 3740688384), total 8607571968)
2019-01-22 16:48:33 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:34 migration status: active (transferred 5010548027, remaining 3435380736), total 8607571968)
2019-01-22 16:48:34 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:35 migration status: active (transferred 5246175649, remaining 3198160896), total 8607571968)
2019-01-22 16:48:35 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:36 migration status: active (transferred 5487444921, remaining 2954059776), total 8607571968)
2019-01-22 16:48:36 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:37 migration status: active (transferred 5660553952, remaining 2778447872), total 8607571968)
2019-01-22 16:48:37 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:38 migration status: active (transferred 5977609173, remaining 2458034176), total 8607571968)
2019-01-22 16:48:38 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:39 migration status: active (transferred 6225147665, remaining 2206580736), total 8607571968)
2019-01-22 16:48:39 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:40 migration status: active (transferred 6537910083, remaining 1883009024), total 8607571968)
2019-01-22 16:48:40 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:42 migration status: active (transferred 6848839715, remaining 1566081024), total 8607571968)
2019-01-22 16:48:42 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:43 migration status: active (transferred 7000180024, remaining 1412694016), total 8607571968)
2019-01-22 16:48:43 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:44 migration status: active (transferred 7162596615, remaining 1248440320), total 8607571968)
2019-01-22 16:48:44 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:45 migration status: active (transferred 7408432109, remaining 998612992), total 8607571968)
2019-01-22 16:48:45 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:46 migration status: active (transferred 7666862833, remaining 736190464), total 8607571968)
2019-01-22 16:48:46 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:47 migration status: active (transferred 7937010611, remaining 437788672), total 8607571968)
2019-01-22 16:48:47 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:48 migration status: active (transferred 8091975445, remaining 256241664), total 8607571968)
2019-01-22 16:48:48 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8253716262, remaining 86441984), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8268958534, remaining 71229440), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8284200806, remaining 56016896), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8299049094, remaining 41197568), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8315079334, remaining 25198592), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 0 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8327676922, remaining 37724160), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 807 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8328702938, remaining 36700160), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 1057 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8333307643, remaining 32100352), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 2179 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8338220292, remaining 27127808), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 3376 overflow 0
2019-01-22 16:48:49 migration status: active (transferred 8344556938, remaining 20779008), total 8607571968)
2019-01-22 16:48:49 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 4920 overflow 0
2019-01-22 16:48:50 migration status: active (transferred 8356581782, remaining 8728576), total 8607571968)
2019-01-22 16:48:50 migration xbzrle cachesize: 1073741824 transferred 0 pages 0 cachemiss 7850 overflow 0
2019-01-22 16:48:50 migration speed: 221.41 MB/s - downtime 16 ms
2019-01-22 16:48:50 migration status: completed
2019-01-22 16:48:54 migration finished successfully (duration 00:00:45)
TASK OK

ending in Kernel Panic.
Screenshot attached.

Stefano Scoppa · Jan 31, 2019

Hi,

any feedback please?

Stefano Scoppa · Feb 1, 2019

so, we did more tests moving a VM between those nodes:

PVE01) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)
PVE02) 16 x Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2 Sockets)
PVE04) 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)
PVE05) 16 x AMD Opteron(TM) Processor 6220 (2 Sockets)
PVE06) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)

root@pve06:~# qm config 226
agent: 1
bootdisk: scsi0
cores: 1
description:
ide2: none,media=cdrom
memory: 2048
name: ubuntu-test-pve
numa: 0
ostype: l26
parent: clean
scsi0: jbod2-pool0-kvm:226/vm-226-disk-0.qcow2,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=5e7513cc-3789-48db-850d-88d2c73e80e5
sockets: 2
tablet: 0
vmgenid: b93f0794-9aba-4285-9de0-8619de741577
root@pve06:~#

if the VM CPU type is: KVM64

moving the VM from INTEL nodes to AMD nodes: KP
moving the VM from AMD nodes to INTEL nodes : KP
moving the VM between AMD nodes: no KP
moving the VM between INTEL nodes: no KP

if the VM CPU type is: HOST

moving the VM from INTEL nodes to AMD nodes: KP
moving the VM from AMD nodes to INTEL nodes : KP
moving the VM between AMD nodes: no KP
moving the VM between INTEL nodes: KP
In fact, we have Xeon(R) CPU X5650 and Xeon(R) CPU E5620, and between the 2 Xeon(R) CPU X5650 we also have a KP probably because one node is configured to let NestedVirtualization, and the other one is not.

At this point, I believe Proxmox is not able to handle a migration between nodes running different CPUs.

Tapio Lehtonen · Feb 1, 2019

At this point, I believe Proxmox is not able to handle a migration between nodes running different CPUs.

That is true. Live migrations at least do not work if cpu differ.
If you do migrations in that situation, use as CPU the default KVM64. Then you can migrate to nodes running different CPU:s.
After migrations are finished, you can change the CPU back to host.

sseidel · Mar 6, 2019

It's a bug. Proxmox team knows about it and won't acknowledge or do something about it. Live migration worked fine in 5.0 and then broke after that.
See https://forum.proxmox.com/threads/live-migration-broken-for-vms-doing-real-work.49380/ and https://bugzilla.proxmox.com/show_bug.cgi?id=1660

Try this:

Code:

qm set **VMID** --args "-cpu 'kvm64,+ssse3,+sse4.1,+sse4.2,+x2apic,+aes,+sep,+movbe,+lahf_lm,+kvm_pv_eoi'"

and shut down and start up your VM.

For me live migration then works even between different CPUs and CPU vendors.

issue on moving Ubuntu Server VMs from one server to another

New Member

Proxmox Staff Member

New Member

Attachments

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

New Member

New Member

Attachments

New Member

Attachments

New Member

Attachments

New Member

Attachments

New Member

Attachments

New Member

New Member

Attachments

New Member

Attachments

New Member

New Member

Active Member

Renowned Member