Proxmox 6.x: Nested VMX virtualization does not support live migration yet

[dan.ger]

Hello,

on a 3 node Ceph Cluster with Proxmox 6, I got following error while live migrating vm with cpu type host and turned on nested virtualization for physical nodes. On Proxmox 5.4.x we have no probs. All physical server are the same. On Proxmox 6.x we got following error:

start migrate command to unix:/run/qemu-server/100.migrate
2019-09-27 13:03:24 migrate uri => unix:/run/qemu-server/100.migrate failed: VM 100 qmp command 'migrate' failed - Nested VMX virtualization does not support live migration yet
2019-09-27 13:03:25 ERROR: online migrate failure - VM 100 qmp command 'migrate' failed - Nested VMX virtualization does not support live migration yet

Workarounds are:
1. Deactivate nested virtualization on nodes
2. Set cpu type of vms from host to kvm64

I found a similar error for Redhat, it seems that they fix it https://bugzilla.redhat.com/show_bug.cgi?id=1689216

I thin it depends on Debian Buster package qemu-kvm, is that right?

PveVersion

proxmox-ve: 6.0-2 (running kernel: 5.0.21-2-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-4.15: 5.4-8
pve-kernel-5.0.21-2-pve: 5.0.21-3
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph: 12.2.12-pve1
ceph-fuse: 12.2.12-pve1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-8
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2


Kind regards,
Daniel

 

[t.lamprecht]

on a 3 node Ceph Cluster with Proxmox 6, I got following error while live migrating vm with cpu type host and turned on nested virtualization for physical nodes. On Proxmox 5.4.x we have no probs. All physical server are the same. On Proxmox 6.x we got following error:

It only worked in 5.4 as long as the VM did not issued VMX/SVM (virtualization) operations (i.e., has itself VMs), else it did not worked there too, just with an silent error.
For this reason the QEMU developers added a hard error for all migrations where the VM CPU has the VMX/SVM flag.
This flag normally only gets added if host nesting is activated (which is still an experimental feature) and CPU type host is used, or the VM explicitly added it via a CPU flag when using other VM CPU types.

I thin it depends on Debian Buster package qemu-kvm, is that right?

Not sure what you mean here, but Proxmox VE does not uses Debians QEMU package.

Also I'm not quite sure what this thread is about, is it more a notice for others or a question for another workaround? Sorry if I just did not read closely enough. We explicitly added a warning in the "pve5to5" upgrade checklist script if nesting was active and VMs with the VMX/SVM flag got detected to allow people seeing this change.

Anyway, while since the 5.0 Kernel the KVM module should be able to migrated VMs with nested guests, QEMU 4.0.0 has not the possibility to do so, this feature only got introduced with QEMU 4.1.0 - so it will be available sooner or later and then work for real - not in some corner cases only.