[SOLVED] Can't boot VM's - VMX disabled

P

popilla20k

New Member
Oct 4, 2023
6
1
3
Hi,

I have issues starting my VM's in PVE 8.

I have enabled VMX, VT-d and VT-x in BIOS configuration, but when I boot all my VM's refuse to boot:
Code: 
stopped: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Then I checked if something was wrong with lscpu:
Code: 
$ lscpu
Architecture: x86_64
  CPU op-mode(s): 32-bit, 64-bit
  Address sizes: 46 bits physical, 48 bits virtual
  Byte Order: Little Endian
CPU(s): 28
  On-line CPU(s) list: 0-27
Vendor ID: GenuineIntel
  BIOS Vendor ID: Intel
  Model name: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz
    BIOS Model name: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz CPU @ 1.7GHz
    BIOS CPU family: 179
    CPU family: 6
    Model: 79
    Thread(s) per core: 2
    Core(s) per socket: 14
    Socket(s): 1
    Stepping: 1
    CPU(s) scaling MHz: 94%
    CPU max MHz: 2500.0000
    CPU min MHz: 1200.0000
    BogoMIPS: 3392.12
    Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor d
                         s_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flex
                         priority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
Virtualization features:
  Virtualization: VT-x
Caches (sum of all):
  L1d: 448 KiB (14 instances)
  L1i: 448 KiB (14 instances)
  L2: 3.5 MiB (14 instances)
  L3: 35 MiB (1 instance)
NUMA:
  NUMA node(s): 1
  NUMA node0 CPU(s): 0-27
Vulnerabilities:
  Gather data sampling: Not affected
  Itlb multihit: KVM: Mitigation: VMX disabled
  L1tf: Mitigation; PTE Inversion
  Mds: Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown: Mitigation; PTI
  Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
  Retbleed: Not affected
  Spec rstack overflow: Not affected
  Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
  Srbds: Not affected
  Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
And discovered that VMX was disabled due to iTLB multihit:
Bash: 
cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit
Code: 
KVM: Mitigation: VMX disabled

The strange thing is that everything was working recently, but I may have changed some bios setup that might be breaking PVE's VMX capability.
I tried already adding to /etc/default/grub file the option kvm.nx_huge_pages:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash kvm.nx_huge_pages=force"
And yes, I ran "update-grub" after saving.

I also tried to turn off all mitigations (mitigations=off), that changes all other mitigations to "vulnerable", but I still get "KVM: Mitigation: VMX disabled"

Any ideas of what might be causing that?

Context: I have installed PVE 8
Code: 
Linux xxxx 6.2.16-14-pve #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-14 (2023-09-19T08:17Z) x86_64 GNU/Linux

Thank you very much.
 
Last edited:
popilla20k said:
I have enabled VMX, VT-d and VT-x in BIOS configuration, but when I boot all my VM's refuse to boot:
Click to expand...
I doubt that VT-x is properly enabled; see below.
popilla20k said:
And discovered that VMX was disabled due to iTLB multihit:
Bash: 
cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit
Code: 
KVM: Mitigation: VMX disabled
Click to expand...
According to the Linux kernel documentation, it means that your system is not vulnerable because VT-x is disabled. Not the other way around, as you suggest.
popilla20k said:
I also tried to turn off all mitigations (mitigations=off), that changes all other mitigations to "vulnerable", but I still get "KVM: Mitigation: VMX disabled"

Any ideas of what might be causing that?
Click to expand...
Because VMX is not disabled by the mitigations but because it is not enabled by your motherboard BIOS; see above.

Can you update your motherboard BIOS and double-check your BIOS settings? Do you also have this problem when you boot the system with a Linux Live CD (like Ubuntu or GParted)? Maybe it's not a Proxmox specific issue and maybe it also affects Windows? Maybe contact your motherboard support provider about this issue?
 
I forgot to mention that I was suspecting that something in BIOS wasn't properly configured, and maybe someone had some idea of what was missing. But yes, I have enabled VMX in the BIOS config, please find attached my current BIOS config with VMX enabled. I also tried several combinations enabling/disabling "Execute Disable Bit" and "Intel TXT Support", but no luck.

As suggested, I booted with a GParted v1.1.0-5 USB that I had around with the same BIOS config and, surprisingly rather than "VMX disabled" it has "Split huge pages", which is the message I was looking for.

I didn't try virtualization in windows yet, so I can't tell if it doesn't work, but looking at the GParted option and the fact that I could make VMX work one week ago, I thought it was something silly I was forgetting.

But now I'm puzzled. Can it be related with a kernel update that enforced VMX to be disabled if the CPU is iTLB vulnerable?

Thanks
 

Attachments

  • Image.jpg
    Image.jpg
    156.8 KB · Views: 20
  • Image (1).jpg
    Image (1).jpg
    177.5 KB · Views: 19
  • Like
Reactions: leesteken
popilla20k said:
As suggested, I booted with a GParted v1.1.0-5 USB that I had around with the same BIOS config and, surprisingly rather than "VMX disabled" it has "Split huge pages", which is the message I was looking for.
Click to expand...
The BIOS settings are probably fine then and it's something Proxmox specific, since another (older) Linux does see VMX enabled (and therefore activates the split-huge-pages mitigation).
popilla20k said:
I didn't try virtualization in windows yet, so I can't tell if it doesn't work, but looking at the GParted option and the fact that I could make VMX work one week ago, I thought it was something silly I was forgetting.
Click to expand...
If you see differences in Linuxes, then I don't think installing Windows is worth the effort.
popilla20k said:
But now I'm puzzled. Can it be related with a kernel update that enforced VMX to be disabled if the CPU is iTLB vulnerable?
Click to expand...
I don't think it works that way, but it is strange and we'll have to take a closer look at Proxmox. Unfortunately, I don't have an Intel system to compare with.
What is the output of cat /proc/cmdline and journalctl -b 0 | egrep -i 'kvm|vmx' and grep -HR '' /etc/modprobe.d/?
 
  • Like
Reactions: popilla20k
These are the outputs:
Code: 
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.2.16-14-pve root=/dev/mapper/pve-root ro quiet

Code: 
# journalctl -b 0 | egrep -i 'kvm|vmx'
Oct 15 13:25:41 xxxxxx pve-guests[1798]: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:42 xxxxxx pvesh[1796]: Starting VM 103 failed: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:42 xxxxxx pve-guests[1867]: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:43 xxxxxx pvesh[1796]: Starting VM 100 failed: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:45 xxxxxx pve-guests[1904]: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:46 xxxxxx pvesh[1796]: Starting VM 102 failed: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.

Code: 
# grep -HR '' /etc/modprobe.d/
/etc/modprobe.d/pve-blacklist.conf:# This file contains a list of modules which are not supported by Proxmox VE 
/etc/modprobe.d/pve-blacklist.conf:
/etc/modprobe.d/pve-blacklist.conf:# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
/etc/modprobe.d/pve-blacklist.conf:blacklist nvidiafb
/etc/modprobe.d/intel-microcode-blacklist.conf:# The microcode module attempts to apply a microcode update when
/etc/modprobe.d/intel-microcode-blacklist.conf:# it autoloads.  This is not always safe, so we block it by default.
/etc/modprobe.d/intel-microcode-blacklist.conf:blacklist microcode
 
popilla20k said:
These are the outputs:
Code: 
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.2.16-14-pve root=/dev/mapper/pve-root ro quiet

Code: 
# journalctl -b 0 | egrep -i 'kvm|vmx'
Oct 15 13:25:41 xxxxxx pve-guests[1798]: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:42 xxxxxx pvesh[1796]: Starting VM 103 failed: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:42 xxxxxx pve-guests[1867]: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:43 xxxxxx pvesh[1796]: Starting VM 100 failed: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:45 xxxxxx pve-guests[1904]: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.
Oct 15 13:25:46 xxxxxx pvesh[1796]: Starting VM 102 failed: KVM virtualisation configured, but not available. Either disable in VM configuration or enable in BIOS.

Code: 
# grep -HR '' /etc/modprobe.d/
/etc/modprobe.d/pve-blacklist.conf:# This file contains a list of modules which are not supported by Proxmox VE
/etc/modprobe.d/pve-blacklist.conf:
/etc/modprobe.d/pve-blacklist.conf:# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
/etc/modprobe.d/pve-blacklist.conf:blacklist nvidiafb
/etc/modprobe.d/intel-microcode-blacklist.conf:# The microcode module attempts to apply a microcode update when
/etc/modprobe.d/intel-microcode-blacklist.conf:# it autoloads.  This is not always safe, so we block it by default.
/etc/modprobe.d/intel-microcode-blacklist.conf:blacklist microcode
Click to expand...
No clue there unfortunately. What is the output of lsmod | grep kvm? Can you do a modprobe kvm and see if that helps?
 
Code: 
# lsmod | grep kvm
#
Code: 
# modprobe kvm
modprobe: FATAL: Module kvm not found in directory /lib/modules/6.2.16-14-pve
 
popilla20k said:
Code: 
# lsmod | grep kvm
#
Code: 
# modprobe kvm
modprobe: FATAL: Module kvm not found in directory /lib/modules/6.2.16-14-pve
Click to expand...
Well that's different from my (AMD) Proxmox system and I find it very weird that there is no kvm module (or built-in).
I also have a newer kernel (6.2.16-15-pve; just updated to 8.0.4). Can you update your Proxmox?
Did a previous apt dist-upgrade fail (or did you mistakenly run apt upgrade)? Is or was your root drive corrupted? Maybe reinstall Proxmox VE 8?
 
Yes, I ran twice apt upgrade, I just checked it in .bash_history. Is it that bad? :oops:

And no, no apt dist-upgrade have ever failed. No drives corrupted
 
popilla20k said:
Yes, I ran twice apt upgrade, I just checked it in .bash_history. Is it that bad? :oops:
Click to expand...
Yes, as many threads here will confirm: never run apt upgrade. Always use the GUI or run apt dist-upgrade as per the manual.
popilla20k said:
And no, no apt dist-upgrade have ever failed. No drives corrupted
Click to expand...
Please run apt update and apt dist-upgrade to see if that installs a new kernel version that just works.
 
Sorry for the delayed response. I've been busy and I couldn't find the time to reinstall proxmox until last week.
It's strange, because I can use now KVM virtualization although I have VMX disabled (no apt upgrade this time :)):
Code: 
  Itlb multihit:         KVM: Mitigation: VMX disabled

Repeating previous requested commands:
Code: 
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.2.16-19-pve root=/dev/mapper/pve-root ro quiet kvm.nx_huge_pages=off

Code: 
# journalctl -b 0 | egrep -i 'kvm|vmx'
Nov 12 20:57:17 astrea kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-19-pve root=/dev/mapper/pve-root ro quiet kvm.nx_huge_pages=off
Nov 12 20:57:17 astrea kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-19-pve root=/dev/mapper/pve-root ro quiet kvm.nx_huge_pages=off

Code: 
# grep -HR '' /etc/modprobe.d/
/etc/modprobe.d/intel-microcode-blacklist.conf:# The microcode module attempts to apply a microcode update when
/etc/modprobe.d/intel-microcode-blacklist.conf:# it autoloads.  This is not always safe, so we block it by default.
/etc/modprobe.d/intel-microcode-blacklist.conf:blacklist microcode
/etc/modprobe.d/pve-blacklist.conf:# This file contains a list of modules which are not supported by Proxmox VE
/etc/modprobe.d/pve-blacklist.conf:
/etc/modprobe.d/pve-blacklist.conf:# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
/etc/modprobe.d/pve-blacklist.conf:blacklist nvidiafb

Code: 
# lsmod | grep kvm
kvm_intel             483328  0
kvm                  1331200  1 kvm_intel
irqbypass              16384  1 kvm

Code: 
# modprobe kvm
#

So... looks like reinstalling did the trick
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom