nested virt - proxmox (debian9(windows srv 16))

ibs0d

Member
Apr 23, 2015
8
0
21
Good day.

For academic purposes nested virtualization is required.
host - proxmox 5.4-3 (CPU:12 x Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (2 Sockets))
1 layer: debian 9 (CPU: host (nested enable))
2 layer: any linux - all goods
2 layer:
  • windows 7-10 (install slow, virtio support installed, boot slow, work slow)
  • windows srv 16 (install slow, after, boot not possible)
what is done: change cpu model for 2 layer - no result

what else can try ?

thx.
 

Stefan_R

Proxmox Staff Member
Staff member
Jun 4, 2019
581
98
28
Vienna
2 Layer nested virtualization is not going to be very fast to begin with. That said, there are assisting technologies available.

For starters, do you have shadow vmcs enabled? (cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs)
Also, did you check that nesting actually works? (grep "vmx" /proc/cpuinfo should show the vmx flag in layer 2/first vm too)

For Windows guests you should also enable Hyper-V enlightenments. Easy way to do this is to run Proxmox as the first guest (layer 2) as well, and then select "Windows" as the second (layer 3) guest OS in there.
 

ibs0d

Member
Apr 23, 2015
8
0
21
2 Layer nested virtualization is not going to be very fast to begin with. That said, there are assisting technologies available.

For starters, do you have shadow vmcs enabled? (cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs)
Also, did you check that nesting actually works? (grep "vmx" /proc/cpuinfo should show the vmx flag in layer 2/first vm too)

For Windows guests you should also enable Hyper-V enlightenments. Easy way to do this is to run Proxmox as the first guest (layer 2) as well, and then select "Windows" as the second (layer 3) guest OS in there.
vmcs: enabled
Hyper-V enlightenments: enabled
 

Stefan_R

Proxmox Staff Member
Staff member
Jun 4, 2019
581
98
28
Vienna
I just did some testing on my machine (Xeon E3-1231 v3) with vmcs enabled and I'm not seeing a particularly big performance drop.

Setup is VE on Host, VE as guest, and Windows 10 as guest on either of these. CPU benchmarking (CBr20) shows a drop of about 7% between regular and nested virtualization.

I would check other vectors as well, depending on how the slowness manifests (CPU, memory, storage, display, etc...). Is your storage fast enough? Are you overcommiting one of the virtualization layers?

For more help I would need more details about your setup (storage type, VM configuration in host and guest).
 

ibs0d

Member
Apr 23, 2015
8
0
21
I just did some testing on my machine (Xeon E3-1231 v3) with vmcs enabled and I'm not seeing a particularly big performance drop.

Setup is VE on Host, VE as guest, and Windows 10 as guest on either of these. CPU benchmarking (CBr20) shows a drop of about 7% between regular and nested virtualization.

I would check other vectors as well, depending on how the slowness manifests (CPU, memory, storage, display, etc...). Is your storage fast enough? Are you overcommiting one of the virtualization layers?

For more help I would need more details about your setup (storage type, VM configuration in host and guest).
storage: ssd

layer 1: proxmox
layer 2: win srv16, all good

update debian 9 from testing repo

layer 1: debian9
layer 2: win srv16, install not possible (os-variant=win2k16)
 

Stefan_R

Proxmox Staff Member
Staff member
Jun 4, 2019
581
98
28
Vienna
That is then an error with your debian guest, I suggest you look into plain QEMU or libvirt documentation.
 

ibs0d

Member
Apr 23, 2015
8
0
21
That is then an error with your debian guest, I suggest you look into plain QEMU or libvirt documentation.
yes

proxmox run winsrv16 with "kvm64,+lahf_lm,+s
ep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer
" but "virsh edit" not support this hv options.
 

ibs0d

Member
Apr 23, 2015
8
0
21
That is then an error with your debian guest, I suggest you look into plain QEMU or libvirt documentation.
solved
i check libvirt log for winsrv guest:
.....
warning: host doesn't support requested feature: CPUID.01H:ECX.vmx
.....​
change cpu model for win2k16 guest to

mode='host-passthrough'

this solve my problem

thx
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!