Issue with nested VM - KVM: entry failed, hardware error 0x7

[Joshua_Roebuck]

Hello Team,
I've been struggling with getting nested virtualization to work in one of my virtual machines. I get an error message when i attempt to start a virtual machine, using qemu-system-x86_64, which immediately fails. Here is the output generated from CLI for the nested VM:


root@vmeve-a:~# qemu-system-x86_64 -machine type=pc,accel=kvm -cpu host -m 4096 -drive file=/opt/unetlab/tmp/0/7961856f-9006-4fa7-8901-1dfa269dd3cd/1/virtioa.qcow2
KVM: entry failed, hardware error 0x7
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000306c3
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 0000ffff
IDT= 00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=ff ff 66 5b 66 83 c4 08 66 5b 66 5e 66 c3 b0 20 e6 20 66 c3 <ea> 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This is the output of LSCPU from within the guest:
root@vmeve-a:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 60
Model name: Intel(R) Core(TM) i5-4570T CPU @ 2.90GHz
Stepping: 3
CPU MHz: 2893.302
BogoMIPS: 5786.60
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 16384K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat umip arch_capabilities

Latest Microcode package is installed in the guest:
root@vmeve-a:~# sudo apt install intel-microcode
Reading package lists... Done
Building dependency tree
Reading state information... Done
intel-microcode is already the newest version (3.20210216.0ubuntu0.16.04.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@vmeve-a:~#

Onwards to my host, here is the output of pveversion below:

root@vm-a:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-5.13: 7.1-4
pve-kernel-helper: 7.1-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph-fuse: 15.2.13-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
root@vm-a:~#

Please help me find a solution for this. The guest VM is EVE-NG and none of my windows VM's hosted in it start anymore.

 

[David Herselman]

We had an almost identical issue with GNS3 VM (v2.2.27) where this VM had been updated to latest available packages in Ubuntu 20.04 (Focal Fossa). After updating the VM to Ubuntu 21.10 (Impish Indri) it's working as expected.

To be clear, we're running the GNS3 VM where we updated Ubuntu to the latest available. The VM is doing nested virtualisation and we're running PVE 7.1 with kernel 5.13.19-1-pve and Ceph Pacific 16.2.6.

Proxmox:
/etc/pve/nodes/kvm1d/qemu-server/107.conf

agent: 1
boot: cdn
bootdisk: scsi0
cores: 2
cpu: host
ide2: none,media=cdrom
localtime: 1
machine: pc-q35-6.0
memory: 4096
name: gns3
net0: virtio=C6:3A:88:A4:16:44,bridge=vmbr0,tag=1
net1: virtio=66:0B:63:A6:0C:A0,bridge=vmbr0,tag=60
numa: 1
onboot: 1
ostype: l26
protection: 1
scsi0: rbd_hdd:vm-107-disk-0,cache=writeback,discard=on,size=21G,ssd=1
scsi1: rbd_hdd:vm-107-disk-1,cache=writeback,discard=on,size=100G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=261344d7-47f1-4924-b0f3-51398b54683a
sockets: 2

When either tailing /opt/gns3/projects/7e280358-ac22-4d04-8ea8-6d1c0b360f3d/project-files/qemu/970bd842-8a2d-44e4-b32f-2bc29c173936/qemu.log (update paths for the VM GNS3 is launching that you are testing with) or manually trying to launch it yields the following warning:

/usr/bin/qemu-system-x86_64 -name B -m 128M -smp cpus=1,sockets=1 -enable-kvm -machine smm=off -boot order=c -drive file=/opt/gns3/projects/7e280358-ac22-4d04-8ea8-6d1c0b360f3d/project-files/qemu/970bd842-8a2d-44e4-b32f-2bc29c173936/hda_disk.qcow2,if=virtio,index=0,media=disk,id=drive0 -uuid 970bd842-8a2d-44e4-b32f-2bc29c173936 -net none -nographic
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
KVM: entry failed, hardware error 0x7
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000206d7
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=04 66 41 eb f1 66 83 c9 ff 66 89 c8 66 5b 66 5e 66 5f 66 c3 <ea> 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

NB: The 'ECX.svm' message is a red herring, this is an Intel Xeon system so it's expected that the AMD svm feature is not available. This warning can be ignored, the problem is the 'KVM: entry failed, hardware error 0x7' message.

Frustratingly KVM features appeared to be available and working in the Ubuntu 20.04 VM with updates as of December 2021:

root@gns3vm:/home/gns3# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

root@gns3vm:/home/gns3# cat /sys/module/kvm_intel/parameters/nested
Y

root@gns3vm:/home/gns3# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'freezer' controller support                     : PASS
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

If it helps others, herewith notes on us upgrading the Ubuntu 20.04 VM (GNS3) to 21.10:

rm -f /etc/apt/sources.list.d/gns3*;
apt-get update; apt-get -y dist-upgrade; apt-get autoremove; apt-get autoclean;
do-release-upgrade;
add-apt-repository ppa:gns3/ppa

Conversion to netplan:
  Network structure:
    eth0   = DHCP on GNS3 server's uplink port
    eth1   = bridged to br0, connecting virtual network clouds to br0 will allow DHCP to the GNS3 virtual lab network and make the IP reachable from our office network.
    virbr0 = NAT environment where GNS3 runs DHCP in 192.168.123.0/24 subnet. You will have outbound access but won't be able to connect to your devices!
 
    ie: Connect 'cloud' in virtual environments to br0. We have disabled virbr0 in our environment.

  apt-get install netplan.io;
  apt purge ifupdown;
  rm -f /etc/network/interface;
  pico /etc/netplan/01-netcfg.yaml
    network:
        version: 2
        renderer: networkd
        ethernets:
            eth0:
                dhcp4: yes
            eth1:
                mtu: 8996
        bridges:
            br0:
                mtu: 8996
                interfaces:
                    - eth1

 

[Joshua_Roebuck]

Hi Team,
Is there anyone else who is experiencing this issue? While I was able to switch to GNS3 using the steps above, i'm not closer to understanding why nested VM's arent able to POST. I'm unable to get any ubuntu versions 16,18 and 20.04 LTS able to nest VM's without running into this issue.

is this a Regression or bug with QEMU pve? Perhaps Proxmox staff could chime in on this one.

As always, I'm grateful for any feedback.

 

[danaelg]

Same here,
I'm not using gns3 VM but I used nested virtualization elsewhere and it broke since upgrade to 5.13.X kernel.

I've seen people trying 5.15.X kernel and it solved the issue (https://forum.proxmox.com/threads/amd-nested-virtualization-not-working-after-7-1-upgrade.100072/).
I'll try this afternoon to roolback to 5.11.X

 

[hrld]

I ran into the same issue on 5.13.19-2 and 5.15.5-1. Rollback to 5.11.22-7 allows me to run nested virtualization again.

 

[fragtion]

Same as problem here. Can't run eve-ng CE (Ubuntu 16.04 image) unless I downgrade pve kernel to 5.11.22-7. problem still not fixed in 5.15! I'm on an Intel i5, not an AMD

 

[Joshua_Roebuck]

I dont think this is a proxmox issue. The developers at Eve-NG decided to continue using 16.04 and 18.04 versions of ubuntu, which are unsupported. I moved to gns server and everything is working as expected. Untill they take security seriously, or thier support framework, i'll stick to GNS server instead of thier suggestion to use a physical server, or change from proxmox to something else.