[SOLVED] KVM VMX Setting for Nested XenServer

[nick80]

I am trying to find a way to set the VMX setting in KVM but the qm.conf format doesn't seem to allow me to pass these specific settings to libvirt/KVM.

Here is the text I am following from a site explaining how to nest a hypervisor and still allow HVM VMs to run inside.

Luckily with KVM we can forward the VMX CPU flag to a guest and as such make it available to Xenserver, for it's HVM mode.

There are a few things to be aware of though:
1 - in libvirt do give the Xenserver VM a good CPU profile (I used Core2duo) and make sure the VMX flag is set on "require"
2 - stock CentOS 7 kernel has a problem with nested virt at the moment, do use a newer kernel[1] (I'm using kernel-ml from elrepo-kernel)
3 - make sure the kvm_intel module is loaded with the option nested=1. For this to happen I reload/rebooted with this in /etc/modprobe.d/kvm-intel.conf:
options kvm-intel nested=1

I have nested KVM working and XenServer installed without issue. I can start PV VMs easily enough but need HVM mode for things like Windows OS. If anyone knows how to get Proxmox to pass this setting It would be a great help. I don't mind setting it globally on the server or inside the VM conf.

UPDATE: As I am researching this, it appears that I might have this flag already by using CPU type of "Host" though XenServer still doesn't see the feature.

Running nested VMX
------------------

The nested VMX feature is disabled by default. It can be enabled by giving
the "nested=1" option to the kvm-intel module.

No modifications are required to user space (qemu). However, qemu's default
emulated CPU type (qemu64) does not list the "VMX" CPU feature, so it must be
explicitly enabled, by giving qemu one of the following options:

-cpu host (emulated CPU has all features of the real CPU)

-cpu qemu64,+vmx (add just the vmx feature to a named CPU type)

UPDATE2: I was able to try to instantiate the VM via the terminal, I added `,+vmx` after the `cpu: host` and it tried to start but threw

warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]

and died. I tried switching to a different CPU with this flag and it failed the same way.

 

[nick80]

I solved my own issue with enough digging. I had to change the options for the kvm-intel module.
The guide has you add one line

options kvm-intel nested=Y

but I actually needed to use

options kvm ignore_msrs=y
options kvm-intel nested=Y ept=Y

 

[nick80]

Thanks for the reply hstende, however my system does indeed support this and it isn't a BIOS setting issue since my Proxmox install can KVM with VTx just fine.

I managed to solve my problem in a reply I posted to myself above. I knew it should work I just couldn't find how to let Proxmox know to send the correct settings when instantiating qemu. I thought I needed to inform qemu of another cpu feature when it actually was passing them all via the setting being on "host". It was just something off in the settings for the kernel module.

 

[nick80]

Why would you delete a comment that could help someone rule out a fix that might not work? (I wasn't talking to myself here!)
For anyone reading this thread, the suggestion was to double check my CPU compatability with the VMX instructions via bios settings and checking CPU features in the terminal with a command such as:

egrep ‘(vmx|svm)’ /proc/cpuinfo

In this case I already knew my hardware was compatible and my question was regarding how to send custom arguments into QEMU at runtime. (I still would like to know if this is possible, but it didn't actually fix my issue when I managed to manually launch QEMU.)

I feel leaving the entire conversation here could be helpful to others. I went down many paths that didn't result in a fix before I hit the winning combination of settings. There happens to be almost nothing that I can find regarding putting Xen in KVM and getting HVM working. Most likely because most sane people have no reason to do this!

 

[hstende]

I had same problem running 64bit vm's on nested esx running in kvm.
I had to set up this in /etc/modprobe.d/kvm.conf on my intel server:
options kvm ignore_msrs=1
options kvm_intel nested=y ept=y
options kvm_intel emulate_invalid_guest_state=0