Nested Virtualization stopped working [PVE 6.1-5]

C

Cookiefamily

Renowned Member
Jan 29, 2020
139
41
68
Germany
Hello,

I just updated from 6.0.7 to 6.1-5 and my guests with nested virtualization no longer start with the following error message:
Code: 
Hyper-V enlightened VMCS (hv-evmcs) requires Hyper-V virtual APIC (hv-vapic)
kvm: kvm_init_vcpu failed: Function not implemented
The processor is set to host,flags=+hv-evmcs and worked like this in the previous version.
(The physical server has a Intel Xeon CPU)

Has anyone had similar issues and was able to solve them?
 
Cookiefamily said:
Hello,

I just updated from 6.0.7 to 6.1-5 and my guests with nested virtualization no longer start with the following error message:
Code: 
Hyper-V enlightened VMCS (hv-evmcs) requires Hyper-V virtual APIC (hv-vapic)
kvm: kvm_init_vcpu failed: Function not implemented
The processor is set to host,flags=+hv-evmcs and worked like this in the previous version.
(The physical server has a Intel Xeon CPU)

Has anyone had similar issues and was able to solve them?
Click to expand...

Never tested before (so can't confirm it used to work on my intel rig), but i get the same issue was well
 
  • Like
Reactions: Cookiefamily
I tried adding hv-vapic as an additional flag in the config file (as that seems to be a solution the error message suggest) but that results in the cpu arguments beeing ignored, the machine reports a Common KVM Processor
 
Just updated and it now seems to work fine for me.

proxmox-ve: 6.1-2 (running kernel: 5.3.13-2-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-2
pve-kernel-helper: 6.1-2
pve-kernel-4.15: 5.4-12
pve-kernel-5.3.13-2-pve: 5.3.13-2
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-17-pve: 4.15.18-43
pve-kernel-4.15.18-16-pve: 4.15.18-41
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-10
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-3
pve-container: 3.0-18
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-4
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1
 
Cookiefamily said:
Hello,

I just updated from 6.0.7 to 6.1-5 and my guests with nested virtualization no longer start with the following error message:
Code: 
Hyper-V enlightened VMCS (hv-evmcs) requires Hyper-V virtual APIC (hv-vapic)
kvm: kvm_init_vcpu failed: Function not implemented
The processor is set to host,flags=+hv-evmcs and worked like this in the previous version.
(The physical server has a Intel Xeon CPU)

Has anyone had similar issues and was able to solve them?
Click to expand...
Wait is that message from inside the level 1 virtual ?

i.e. the Proxmox in the proxmox ?
 
Its not the proxmox in the proxmox, that is the "parent" proxmox when I try to start a "children" proxmox.
The conf file still exists. What do you mean by "modules are loaded"?
 
https://pve.proxmox.com/wiki/Nested_Virtualization

About half way down it says to either reload the module or reboot,

to check if is enabled do ("kvm_intel" for intel cpu, "kvm_amd" for AMD)

root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
N



N means it's not, to enable ("kvm-intel" for intel):

# echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf


(or "kvm-amd" for AMD, note the 1 instead of Y):

# echo "options kvm-amd nested=1" > /etc/modprobe.d/kvm-amd.conf


and reboot or reload the kernel modul

modprobe -r kvm_intel
modprobe kvm_intel


check again

root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
Y
Click to expand...
 
Just tried booting an older Kernel, did not help.
What I did notice though when I try to boot with the parameters cpu: host,flags=+hv-evmcs;+hv-vapic:

Code: 
vm 109 - unable to parse value of 'cpu' - format error
flags: value does not match the regex pattern

Did I mess up the syntax of the flags? I don't see the mistake
 
i've just checked my config after adding flags for "hv-evmcs" and "aes" and it reads

Code: 
cpu: host,flags=+hv-evmcs;+aes

I didn't see an option in the GUI for HV-VACPI
 
add below to your VM configuration file and then boot the VM:

args: -cpu host,+svm


Complete configuration file would look something like this as per options you choose:


bootdisk: scsi0
cores: 8
args: -cpu host,+svm
cpu: kvm64,flags=+hv-evmcs;+aes
ide2: local:iso/proxmox-ve_6.1-1.iso,media=cdrom
memory: 16384
name: vm2
net0: virtio=BA:CA:13:CE:4A:E9,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: Pve8:vm-5001-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=6bca378f-dc6f-41b4-9b13-e526da7702a0
sockets: 1
vmgenid: f3a42be1-b6c9-4979-969b-eac838aa6c9f
 
  • Like
Reactions: ivan2019, zhivko, z3t4 and 2 others
chrispage1 said:
Sorry to dig this one up, but what exactly is it doing behind the scenes. Is it setting the CPU model to host rather than the chosen CPU model?

Thanks!
Click to expand...

So for anyone searching for this, the solution effectively just overwrites the chosen CPU model to use the host CPU directly. This will introduce migration problems etc if you're machines aren't running the same CPU's with the same updates. You're best off writing your own CPU model (https://pve.proxmox.com/pve-docs/cp...text=DESCRIPTION,for example custom CPU flags).)

The processors we run are Intel Broadwell or newer. I was able to take the various flags from the CPU's where I knew support was available, determine the minimum number of physical bits, and then create my custom definition based on supported flags, giving maximum functionality and performance to the VM.

Code: 
cpu-model: my-cpu-definition
  flags +fpu;+vme;+de;+pse;+tsc;+msr;+pae;+mce;+cx8;+apic;+sep;+mtrr;+pge;+mca;+cmov;+pat;+pse36;+clflush;+mmx;+fxsr;+sse;+sse2;+ss;+syscall;+nx;+pdpe1gb;+rdtscp;+lm;+pni;+pclmulqdq;+vmx;+ssse3;+fma;+cx16;+pdcm;+pcid;+spec-ctrl;+md-clear;+sse4_1;+sse4_2;+x2apic;+movbe;+popcnt;+aes;+xsave;+avx;+f16c;+rdrand;+hypervisor;+lahf_lm;+abm;+3dnowprefetch;+ssbd;+ibrs;+ibpb;+stibp;+fsgsbase;+tsc_adjust;+bmi1;+hle;+avx2;+smep;+bmi2;+erms;+invpcid;+rtm;+rdseed;+adx;+smap;+xsaveopt;+arat;+umip
  phys-bits 46
  hidden 0
  hv-vendor-id proxmox
  reported-model kvm64

This enables all the flags compatible with all my CPUIn doing this, I have been able to increase my benchmark scores using the default CPU model (using yabs.sh which runs a Geekbench 6) from 557 single core to 920!
 
  • Like
Reactions: zhivko
You must log in or register to reply here.
Top Bottom