Nested Virtualization stopped working [PVE 6.1-5]

C

Cookiefamily

Well-Known Member
Jan 29, 2020
136
40
48
Germany
Hello,

I just updated from 6.0.7 to 6.1-5 and my guests with nested virtualization no longer start with the following error message:
Code: 
Hyper-V enlightened VMCS (hv-evmcs) requires Hyper-V virtual APIC (hv-vapic)
kvm: kvm_init_vcpu failed: Function not implemented
The processor is set to host,flags=+hv-evmcs and worked like this in the previous version.
(The physical server has a Intel Xeon CPU)

Has anyone had similar issues and was able to solve them?
 
Cookiefamily said:
Hello,

I just updated from 6.0.7 to 6.1-5 and my guests with nested virtualization no longer start with the following error message:
Code: 
Hyper-V enlightened VMCS (hv-evmcs) requires Hyper-V virtual APIC (hv-vapic)
kvm: kvm_init_vcpu failed: Function not implemented
The processor is set to host,flags=+hv-evmcs and worked like this in the previous version.
(The physical server has a Intel Xeon CPU)

Has anyone had similar issues and was able to solve them?
Click to expand...

Never tested before (so can't confirm it used to work on my intel rig), but i get the same issue was well
 
  • Like
Reactions: Cookiefamily
I tried adding hv-vapic as an additional flag in the config file (as that seems to be a solution the error message suggest) but that results in the cpu arguments beeing ignored, the machine reports a Common KVM Processor
 
Just updated and it now seems to work fine for me.

proxmox-ve: 6.1-2 (running kernel: 5.3.13-2-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-2
pve-kernel-helper: 6.1-2
pve-kernel-4.15: 5.4-12
pve-kernel-5.3.13-2-pve: 5.3.13-2
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-17-pve: 4.15.18-43
pve-kernel-4.15.18-16-pve: 4.15.18-41
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-10
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-3
pve-container: 3.0-18
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-4
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1
 
Cookiefamily said:
Hello,

I just updated from 6.0.7 to 6.1-5 and my guests with nested virtualization no longer start with the following error message:
Code: 
Hyper-V enlightened VMCS (hv-evmcs) requires Hyper-V virtual APIC (hv-vapic)
kvm: kvm_init_vcpu failed: Function not implemented
The processor is set to host,flags=+hv-evmcs and worked like this in the previous version.
(The physical server has a Intel Xeon CPU)

Has anyone had similar issues and was able to solve them?
Click to expand...
Wait is that message from inside the level 1 virtual ?

i.e. the Proxmox in the proxmox ?
 
Its not the proxmox in the proxmox, that is the "parent" proxmox when I try to start a "children" proxmox.
The conf file still exists. What do you mean by "modules are loaded"?
 
https://pve.proxmox.com/wiki/Nested_Virtualization

About half way down it says to either reload the module or reboot,

to check if is enabled do ("kvm_intel" for intel cpu, "kvm_amd" for AMD)

root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
N



N means it's not, to enable ("kvm-intel" for intel):

# echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf


(or "kvm-amd" for AMD, note the 1 instead of Y):

# echo "options kvm-amd nested=1" > /etc/modprobe.d/kvm-amd.conf


and reboot or reload the kernel modul

modprobe -r kvm_intel
modprobe kvm_intel


check again

root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
Y
Click to expand...
 
Just tried booting an older Kernel, did not help.
What I did notice though when I try to boot with the parameters cpu: host,flags=+hv-evmcs;+hv-vapic:

Code: 
vm 109 - unable to parse value of 'cpu' - format error
flags: value does not match the regex pattern

Did I mess up the syntax of the flags? I don't see the mistake
 
i've just checked my config after adding flags for "hv-evmcs" and "aes" and it reads

Code: 
cpu: host,flags=+hv-evmcs;+aes

I didn't see an option in the GUI for HV-VACPI
 
add below to your VM configuration file and then boot the VM:

args: -cpu host,+svm


Complete configuration file would look something like this as per options you choose:


bootdisk: scsi0
cores: 8
args: -cpu host,+svm
cpu: kvm64,flags=+hv-evmcs;+aes
ide2: local:iso/proxmox-ve_6.1-1.iso,media=cdrom
memory: 16384
name: vm2
net0: virtio=BA:CA:13:CE:4A:E9,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: Pve8:vm-5001-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=6bca378f-dc6f-41b4-9b13-e526da7702a0
sockets: 1
vmgenid: f3a42be1-b6c9-4979-969b-eac838aa6c9f
 
  • Like
Reactions: zhivko, z3t4, TOMO1982 and 1 other person
chrispage1 said:
Sorry to dig this one up, but what exactly is it doing behind the scenes. Is it setting the CPU model to host rather than the chosen CPU model?

Thanks!
Click to expand...

So for anyone searching for this, the solution effectively just overwrites the chosen CPU model to use the host CPU directly. This will introduce migration problems etc if you're machines aren't running the same CPU's with the same updates. You're best off writing your own CPU model (https://pve.proxmox.com/pve-docs/cp...text=DESCRIPTION,for example custom CPU flags).)

The processors we run are Intel Broadwell or newer. I was able to take the various flags from the CPU's where I knew support was available, determine the minimum number of physical bits, and then create my custom definition based on supported flags, giving maximum functionality and performance to the VM.

Code: 
cpu-model: my-cpu-definition
  flags +fpu;+vme;+de;+pse;+tsc;+msr;+pae;+mce;+cx8;+apic;+sep;+mtrr;+pge;+mca;+cmov;+pat;+pse36;+clflush;+mmx;+fxsr;+sse;+sse2;+ss;+syscall;+nx;+pdpe1gb;+rdtscp;+lm;+pni;+pclmulqdq;+vmx;+ssse3;+fma;+cx16;+pdcm;+pcid;+spec-ctrl;+md-clear;+sse4_1;+sse4_2;+x2apic;+movbe;+popcnt;+aes;+xsave;+avx;+f16c;+rdrand;+hypervisor;+lahf_lm;+abm;+3dnowprefetch;+ssbd;+ibrs;+ibpb;+stibp;+fsgsbase;+tsc_adjust;+bmi1;+hle;+avx2;+smep;+bmi2;+erms;+invpcid;+rtm;+rdseed;+adx;+smap;+xsaveopt;+arat;+umip
  phys-bits 46
  hidden 0
  hv-vendor-id proxmox
  reported-model kvm64

This enables all the flags compatible with all my CPUIn doing this, I have been able to increase my benchmark scores using the default CPU model (using yabs.sh which runs a Geekbench 6) from 557 single core to 920!
 
  • Like
Reactions: zhivko
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back