The Reasons for poor performance of Windows when the CPU type is host

@t.lamprecht Your initial response here seemed somewhat dismissive and focused on the old hardware that was in use, understandable at first given the claimed impact, but this has been demonstrated on the latest Intel desktop-class processors as well now and by a number of different users.

Could you please review the information that’s been gathered here since the initial post and provide an updated recommendation for users? Right now the available presets leave us with a choice between working nested virtualization (“host”) and useable performance (“x86_64-v*”), with the next available option seeming to be each user going through the process of creating a custom CPU type that is a close match for “host” while working around this issue.

I suspect that you and the Proxmox team would prefer to support a simpler best practice than any user with a Windows guest needing to fiddle with custom CPU types.
 
Last edited:
  • Like
Reactions: sebas1221 and blturner
Please note that Proxmox VE already defaults to the x86-64-v2-AES for any VM created through the web UI's "Create VM" wizard since Proxmox VE 8.0 released two years ago.

Exposing the relevant flags for nested virtualization in the UI might be done, especially if it's only a small set of flags, we got the infrastructure for that already, so it would not be that much work.
And FWIW, there was also a proof of concept patches for a UI integration for the custom CPU models, which would lower the barrier significantly, which probably makes sense to have in any way. They were not picked up due to relatively low demand and might need some rebasing though.

Feel free to open an enhancement report in our Bugzilla instance and link this post here, I'd focus the request on allowing one to enable nested virtualization for VMs through the web UI, that is your core goal after all and leaves enough freedom for choosing a fitting implementation.
sbrocket said:
simpler best practice than any user with a Windows guest needing to fiddle with custom CPU types.
Click to expand...
Well, to be fair, it's for any Windows guest that also needs nested virtualization, all others work just fine and performant with the default CPU type. Don't get me wrong, as written above, we're open with improving the status quo for those preferring the web UI, but please avoid adding overblown statements, they do not really help to advance one cause, and might even do the opposite...
 
  • Like
Reactions: weehooey-bh
t.lamprecht said:
Well, to be fair, it's for any Windows guest that also needs nested virtualization, all others work just fine and performant with the default CPU type. Don't get me wrong, as written above, we're open with improving the status quo for those preferring the web UI, but please avoid adding overblown statements, they do not really help to advance one cause, and might even do the opposite...
Click to expand...

I have been in your shoes many times and understand that sentiment, but I think that statement was only minorly overblown. Nested virtualization with a Windows guest is not at all an exotic requirement since it’s required for WSL. I don’t have any kind of usage stats, but I’d guess that given the already technical nature of many Proxmox users that using WSL within a Windows guest is far from uncommon. And the best available recommendations for a Windows guest with nested virt (on the Wiki, a community effort to be sure but still the best available) now point you in a direction that leads to clear poor performance, and it seems unclear what to update those best practices to.

Plus, even if Proxmox’s default is not “host”, the fact that “host” is the currently recommended best practice for some use cases and seems like a reasonable setting for most use cases where it isn’t clearly contraindicated does, I bet, lead people to use it whether they strictly need nested virtualization or not. The fix is certainly simpler if you don’t need nested virtualization, that’s true, but discovering the link between CPU type and poor performance was not easy and I would bet generates plenty of support noise on its own.

In any case the motivation was to call out that you might be looking at a burgeoning support headache if this advice proliferates and people make predictable errors in creating custom CPU types, and may want to get ahead of it. I’m not under any delusions that my personal usage of Proxmox is important to anyone, nor am I worried about my own ability to figure that out given some time and effort. :)

I’m sure my prominent “New Member” title makes for some warranted default skepticism, whatever the content of the argument, but I’m not aiming to exaggerate here.
 
Last edited:
  • Like
Reactions: roggeb
I've run another test after creating a custom CPU type which is host minus the specific two flags that the OP @Kobayashi_Bairuo claimed were the source of the issue, md-clear and flush-l1d:

Code: 
❯ cat /etc/pve/virtual-guest/cpu-models.conf
cpu-model: host-windows-workaround
    flags -md-clear;-flush-l1d
    phys-bits host
    hidden 0
    hv-vendor-id proxmox
    reported-model host

Unfortunately, disabling only these two flags does not result in the same improvement in my Windows guest's performance that using x86_64-v3 does, so I suspect that something in the details of OP's analysis was incorrect. (But don't misunderstand me, I definitely appreciate OP's findings!)

I'm far from certain, but I would guess that the source of the error is from attempting to directly compare the CPU flag names from /proc/cpuinfo with QEMU's supported CPU flag names as reported by qemu-system-x86_64 -cpu help, which seem to have slightly different spellings in many cases. For example, cpuinfo reports md_clear, but QEMU spells it md-clear . I manually compared the flags reported by cpuinfo for host, Skylake-Client-v4, and x86_64-v3, but could not clearly identify corresponding QEMU flags for a handful of my i7-14700k's cpuinfo-reported flags (ibrs_enhanced tpr_shadow flexpriority ept ept_ad ospke) that were not in Skylake-Client-v4, and that was before attempting to match up all the vmx flags.

A reasonably safe and proper fix that keeps nested virtualization working while fixing performance is still unclear to me at this point, and I don't love the idea of fiddling with individual CPU flags that I don't individually understand the impact of. Skylake-Client-v4 seems to be a closer fit for my host CPU's flags than x86_64-v3 and does still have good performance, so I guess my next step is to attempt to add on the right nested virtualization flags and see what that does.

If anyone knows...should I worry about trying to match my host's vmx flags closely, or is something simpler like Skylake-Client-v4,+vmx likely to be ok? There are quite a few vmx flags reported by cpuinfo for the host CPU:

Code: 
processor       : 27
vendor_id       : GenuineIntel
cpu family      : 6
model           : 183
model name      : Intel(R) Core(TM) i7-14700K
stepping        : 1
microcode       : 0x12f
cpu MHz         : 4300.014
cache size      : 33792 KB
physical id     : 0
siblings        : 28
core id         : 43
cpu cores       : 20
apicid          : 86
initial apicid  : 86
fpu             : yes
fpu_exception   : yes
cpuid level     : 32
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities
vmx flags       : vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid ple shadow_vmcs ept_mode_based_exec tsc_scaling usr_wait_pause
bugs            : spectre_v1 spectre_v2 spec_store_bypass swapgs eibrs_pbrsb rfds bhi
bogomips        : 6835.20
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:
 
Last edited:
sbrocket said:
I've run another test after creating a custom CPU type which is host minus the specific two flags that the OP @Kobayashi_Bairuo claimed were the source of the issue, md-clear and flush-l1d:

Code: 
❯ cat /etc/pve/virtual-guest/cpu-models.conf
cpu-model: host-windows-workaround
    flags -md-clear;-flush-l1d
    phys-bits host
    hidden 0
    hv-vendor-id proxmox
    reported-model host

Unfortunately, disabling only these two flags does not result in the same improvement in my Windows guest's performance that using x86_64-v3 does, so I suspect that something in the details of OP's analysis was incorrect. (But don't misunderstand me, I definitely appreciate OP's findings!)
Click to expand...

Back then, I did exactly the same and made the same observation.
I also fiddled around with other flags and combinations, but without success and rather quickly stopped going down this rabbit hole any further...
 
Heh, I’d love to stop going down this rabbit hole, but I still want to solve for adding nested virtualization onto the next closest preset (Skylake-Client-v4, as far as I can tell) now. I’ll have to give that a try later tonight.

My initial claim that nested virtualization probably has significant usage with Windows guests is seeming less reasonable than I initially thought based on these responses, I’m not sure now, but I didn’t suspect WSL usage in a guest was really that odd of a requirement. Perhaps it is? In my case I’m aiming to use this guest for a thin client setup at my 3d printing+tinkering workbench, replacing a Windows laptop that was a temporary solution but that I don’t want to keep there permanently.

Perhaps it’s time to reevaluate whether Windows is really the best option for this setup of mine, instead of simply shifting my existing setup into a VM one-for-one. I admit my preference for Windows here comes from a probably outdated heuristic of mine, which is that I’m likely to end up spending more time tinkering with or trying to work around the practical limitations of using desktop Linux. For this particular system, I’d like to prioritize it just working when I want to use it rather than be a time sink of its own. But clearly I’m not meeting that goal currently…after all here I am now sinking time into getting my WSL/Linux shell working within Windows. I use Linux in various forms plenty and clearly prefer it on the command line, but I’m not an open-source purist and I tend to weigh the overall experience over simply OSS vs not.
 
Last edited:
I can't access the blog, I'm assuming the method adds flags to allow nested virtualization to the x86_64-v* types? Can someone re share the info please?
 
Archive.org helps here.
Their conclusion was to add this to /etc/pve/virtual-guest/cpu-models.conf
Code: 
cpu-model: windows-host
    flags +ss;+ssbd;+stibp;+pdpe1gb;+tsc_adjust;+pdcm;+vmx;+umip;+pcid
    phys-bits host
    hidden 0
    hv-vendor-id proxmox
    reported-model IvyBridge-IBRS
 
Last edited:
  • Like
Reactions: Johannes S and UdoB
Wrote a small shell script to detect proper cpu configuration:


Bash: 
#!/usr/bin/env bash
# cputest.sh — Detect x86-64 micro-arch levels (v2/v3/v4) and show a tidy summary.
# Options:
#   --no-color   disable ANSI colors
#   --full       also print the full "common flags across all cores" list

set -euo pipefail

# ------------------------ Options & color ------------------------
USE_COLOR=1
SHOW_FULL=0
for arg in "$@"; do
  case "$arg" in
    --no-color) USE_COLOR=0 ;;
    --full)     SHOW_FULL=1 ;;
  esac
done

if [[ ! -t 1 ]]; then USE_COLOR=0; fi

if (( USE_COLOR )); then
  C_ORANGE=$'\e[38;5;208m'
  C_GRAY=$'\e[90m'
  C_GREEN=$'\e[32m'
  C_CYAN=$'\e[36m'
  C_RESET=$'\e[0m'
  C_BOLD=$'\e[1m'
else
  C_ORANGE=""; C_GRAY=""; C_GREEN=""; C_CYAN=""; C_RESET=""; C_BOLD=""
fi

# ------------------------ Header ------------------------
print_header() {
cat <<'EOF'
               P R O X M O X   V M   C P U   C H E C K

EOF
}

print_header
echo

# ------------------------ Collect flags ------------------------
mapfile -t all_flags < <(grep -E '^flags\s*:' /proc/cpuinfo | cut -d: -f2- | sed 's/^[[:space:]]*//')
if ((${#all_flags[@]}==0)); then
  echo "Could not read CPU flags from /proc/cpuinfo." >&2
  exit 1
fi

# Build intersection of flags across all cores
read -ra base <<< "${all_flags[0]}"
declare -A present
for f in "${base[@]}"; do present["$f"]=1; done
for ((i=1;i<${#all_flags[@]};i++)); do
  declare -A cur=()
  read -ra tmp <<< "${all_flags[$i]}"
  for f in "${tmp[@]}"; do cur["$f"]=1; done
  for f in "${!present[@]}"; do
    [[ -n "${cur[$f]:-}" ]] || unset 'present[$f]'
  done
done

has() { [[ -n "${present[$1]:-}" ]]; }
require_all() { for f in "$@"; do has "$f" || return 1; done; }

# ------------------------ Levels (psABI/GCC aligned) ------------------------
v2_req=(cx16 lahf_lm popcnt sse4_1 sse4_2 ssse3)
v3_req=(avx avx2 bmi1 bmi2 f16c fma movbe xsave)
has_lzcnt() { has lzcnt || has abm; }
v4_req=(avx512f avx512dq avx512cd avx512bw avx512vl)

level=1
if require_all "${v2_req[@]}"; then
  level=2
  if require_all "${v3_req[@]}" && has_lzcnt; then
    level=3
    if require_all "${v4_req[@]}"; then
      level=4
    fi
  fi
fi

cpu_model=$(grep -m1 '^model name' /proc/cpuinfo | cut -d: -f2- | sed 's/^[[:space:]]*//')

# ------------------------ Pretty printing helpers ------------------------
comma_join() { local IFS=", "; echo "$*"; }
print_cols() { # prints words in N columns (default 4)
  local COLS=${1:-4}; shift
  local i=0
  for w in "$@"; do
    printf "  %-14s" "$w"
    (( ++i % COLS == 0 )) && printf "\n"
  done
  (( i % COLS )) && printf "\n"
}

# ------------------------ Output ------------------------
echo "${C_BOLD}CPU:${C_RESET} $cpu_model"
case $level in
  4) echo "${C_BOLD}Detected level:${C_RESET} ${C_ORANGE}x86-64-v4${C_RESET}" ;;
  3) echo "${C_BOLD}Detected level:${C_RESET} ${C_GREEN}x86-64-v3${C_RESET}" ;;
  2) echo "${C_BOLD}Detected level:${C_RESET} ${C_CYAN}x86-64-v2${C_RESET}" ;;
  1) echo "${C_BOLD}Detected level:${C_RESET} x86-64-v1 baseline" ;;
esac
echo

echo "${C_BOLD}Key capabilities:${C_RESET}"
# Show a concise, readable summary
summary_flags=(avx avx2 fma f16c bmi1 bmi2 movbe lzcnt aes pclmulqdq sha_ni adx invpcid fsgsbase rdseed rtm hle)
have=(); for f in "${summary_flags[@]}"; do [[ -n "${present[$f]:-}" ]] && have+=("$f"); done
print_cols 6 "${have[@]}"
echo

echo "${C_BOLD}Level criteria check:${C_RESET}"
printf "  v2 requires: "; print_cols 6 "${v2_req[@]}"
printf "  v3 adds:     "; print_cols 6 "${v3_req[@]}" lzcnt
printf "  v4 adds:     "; print_cols 6 "${v4_req[@]}"
echo

if (( SHOW_FULL )); then
  echo "${C_BOLD}Common flags across all cores:${C_RESET}"
  mapfile -t all_present_sorted < <(printf '%s\n' "${!present[@]}" | sort)
  # print in 6 columns for compactness
  print_cols 6 "${all_present_sorted[@]}"
fi

exit 0
 
  • Like
Reactions: chrcoluk, Johannes S and UdoB
# ./detect-cpu-flags.sh
P R O X M O X V M C P U C H E C K


CPU: AMD Ryzen Threadripper 3960X 24-Core Processor
Detected level: x86-64-v3

Key capabilities:
avx avx2 fma f16c bmi1 bmi2
movbe aes pclmulqdq sha_ni adx fsgsbase
rdseed

Level criteria check:
v2 requires: cx16 lahf_lm popcnt sse4_1 sse4_2 ssse3

GRUB:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=1 iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 kvm_amd.nested=1 kvm_amd.sev=1 nmi_watchdog=0 nofb nomodeset video=efifb: off video=vesafb: off rd.driver.pre=vfio-pci pci-stub.ids=1043:87cb,1022:149c vfio-pci.ids=10de:2204,10de:1aef,10de:1c82,10de:0fb9,8086:10c9,1bb1:5013 modprobe.blacklist=radeon,amdgpu,fglrx,nouveau
,nvidiafb,nvidia,nvidia_drm,nv,snd-pscp,snd_hda_intel hugepagesz=1G hugepages=48 clocksource=tsc tsc=reliable trace_clock=local mitigations=off"

Currently running the VM with the following config:
Code: 
agent: 1
balloon: 0
bios: ovmf
boot: order=sata2;hostpci0
cores: 16
cpu: host,flags=-ibpb;-virt-ssbd;-amd-ssbd;+amd-no-ssb;+pdpe1gb;+aes
cpuunits: 262143
efidisk0: NVME_SMS:vm-113-disk-1,size=4M
hostpci0: 0000:22:00,pcie=1,rombar=0
hostpci1: 0000:01:00,pcie=1,x-vga=1
hostpci2: 0000:44:00.1,pcie=1,rombar=0
hostpci3: 44:00.3,pcie=1,rombar=0
hostpci4: 0000:21:00,pcie=1,rombar=0
hugepages: 1024
machine: pc-q35
memory: 49152
name: Windows11
net0: virtio=86:49:17:1D:EA:28,bridge=vmbr0,tag=555
numa: 1
onboot: 1
ostype: win10
sata2: none,media=cdrom
scsi2: /dev/disk/by-id/ata-ST5000DM000-1FK178_W4J04EPK,backup=0,cache=writeback,iothread=1,size=4883770584K
scsi3: /dev/disk/by-id/ata-ST12000NM000J-2TY103_ZRT0EG3Z,backup=0,cache=writeback,iothread=1,size=11176G
scsihw: virtio-scsi-single
smbios1: uuid=eeeb9398-9d7e-48b0-b27c-8315717f96e2
sockets: 1
startup: order=2,up=30
usb0: host=3-2,usb3=1
vga: none
virtio0: NVME_SMS:vm-113-disk-0,discard=on,iothread=1,size=40G
vmgenid: b22af7d5-adb4-4dee-a075-a861dec43616


Sporadic latency spikes, trouble with real-time audio (pops, clicks).
Any suggestions for improvement?
 
I had noticed something similar also. My 5600G as host CPU, the VM's are more performant using EPYC as cputype instead of host. The explanation discovered makes sense.

On Intel systems I have started using 'x86-64-v*' CPU type.
 
Last edited:
Landed in this thread after trying to use WSL on v2-aes and experiencing exact same issues described all over here. I need WSL to migrate huge amounts of data with rsync from a remote linux host to windows 2025 and wsl with rsync seemed like by far the best option given the amount of data and a large possibility of broken pipes during transfer. As it stands this is a staging pre-production environment so enabling host for the time being and disabling it later is an option for me, but still renders VM nearly useless during this time. Personally I don't believe people using nested virtualization are in such a minority they are to be ignored, but that's just my thinking.
 
You must log in or register to reply here.
Top Bottom