Incorrect supported CPU flags

[eider]

Recent update to pve-manager includes ability to query supported flags via call to qemu --cpu help, however utilizing this endpoint is in vain and incorrect as it will almost universally always present incorrect list.

For example, for this Intel CPU:

root@riko:~# lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             39 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      8
  On-line CPU(s) list:       0-7
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Xeon(R) CPU E3-1275 v6 @ 3.80GHz
    CPU family:              6
    Model:                   158
    Thread(s) per core:      2
    Core(s) per socket:      4
    Socket(s):               1
    Stepping:                9
    CPU(s) scaling MHz:      45%
    CPU max MHz:             3800.0000
    CPU min MHz:             800.0000
    BogoMIPS:                7599.80
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopol
                             ogy nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm a
                             bm 3dnowprefetch cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec x
                             getbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi md_clear flush_l1d arch_capabilities
Virtualization features:
  Virtualization:            VT-x

QEMU claims it supports amd-ssbd and 3dnow:

Recognized CPUID flags:
  3dnow 3dnowext 3dnowprefetch abm ace2 ace2-en acpi adx aes amd-no-ssb
  amd-psfd amd-ssbd amd-stibp amx-avx512 amx-bf16 amx-bf16-alias
  amx-complex amx-complex-alias amx-fp16 amx-fp16-alias amx-fp8 amx-int8
  amx-int8-alias amx-movrs amx-tf32 amx-tile apic apx-nci-ndd-nf apxf arat
  arch-capabilities arch-lbr auto-ibrs avic avx avx-ifma avx-ne-convert
  avx-vnni avx-vnni-int16 avx-vnni-int8 avx10 avx10-128 avx10-256 avx10-512
  avx10-vnni-int avx2 avx512-4fmaps avx512-4vnniw avx512-bf16 avx512-fp16
  avx512-vp2intersect avx512-vpopcntdq avx512bitalg avx512bw avx512cd
  avx512dq avx512er avx512f avx512ifma avx512pf avx512vbmi avx512vbmi2
  avx512vl avx512vnni bhi-ctrl bhi-no bmi1 bmi2 bus-lock-detect cet-ibt
  cet-ss cid cldemote clflush clflushopt clwb clzero cmov cmp-legacy
  cmpccxadd core-capability cr8legacy cx16 cx8 dca ddpd-u de decodeassists
  ds ds-cpl dtes64 eraps erms est extapic f16c fb-clear fbsdp-no
  fdp-excptn-only flush-l1d flushbyasid fma fma4 fpu fred fs-gs-base-ns
  fsgsbase fsrc fsrm fsrs full-width-write fxsr fxsr-opt fzrm gds-no gfni
  gmet hle ht hypervisor ia64 ibpb ibpb-brtype ibrs ibrs-all ibs intel-psfd
  intel-pt intel-pt-lip invpcid invtsc ipred-ctrl its-no kvm-asyncpf
  kvm-asyncpf-int kvm-asyncpf-vmexit kvm-hint-dedicated kvm-mmu
  kvm-msi-ext-dest-id kvm-nopiodelay kvm-poll-control kvm-pv-eoi kvm-pv-ipi
  kvm-pv-sched-yield kvm-pv-tlb-flush kvm-pv-unhalt kvm-steal-time kvmclock
  kvmclock kvmclock-stable-bit la57 lahf-lm lam lbrv
  lfence-always-serializing lkgs lm lwp mca mcdt-no mce md-clear mds-no
  misalignsse mmx mmxext monitor movbe movdir64b movdiri movrs mpx msr
  msr-imm mtrr no-nested-data-bp nodeid-msr npt nrip-save null-sel-clr-base
  nx osvw overflow-recov pae pat pause-filter pbe pbrsb-no pcid pclmulqdq
  pcommit pdcm pdpe1gb perfctr-core perfctr-nb perfmon-v2 pfthreshold pge
  phe phe-en pks pku pmm pmm-en pn pni popcnt prefetchi prefetchiti
  pschange-mc-no psdp-no pse pse36 rdctl-no rdpid rdrand rdseed rdtscp
  rfds-clear rfds-no rrsba-ctrl rsba rtm sbdr-ssdp-no sbpb sep serialize
  sgx sgx-aex-notify sgx-debug sgx-edeccssa sgx-exinfo sgx-kss sgx-mode64
  sgx-provisionkey sgx-tokenkey sgx1 sgx2 sgxlc sha-ni sha512 skinit
  skip-l1dfl-vmentry sm3 sm4 smap smep smx spec-ctrl split-lock-detect
  srso-no srso-user-kernel-no ss ssb-no ssbd sse sse2 sse4.1 sse4.2 sse4a
  ssse3 stibp stibp-always-on succor svm svm-lock svme-addr-chk syscall
  taa-no tbm tce tm tm2 topoext tsa-l1-no tsa-sq-no tsc tsc-adjust
  tsc-deadline tsc-scale tsx-ctrl tsx-ldtrk umip v-vmsave-vmload vaes
  verw-clear vgif virt-ssbd vmcb-clean vme vmx vmx-activity-hlt
  vmx-activity-shutdown vmx-activity-wait-sipi vmx-any-errcode
  vmx-apicv-register vmx-apicv-vid vmx-apicv-x2apic vmx-apicv-xapic
  vmx-cr3-load-noexit vmx-cr3-store-noexit vmx-cr8-load-exit
  vmx-cr8-store-exit vmx-desc-exit vmx-enable-user-wait-pause
  vmx-encls-exit vmx-entry-ia32e-mode vmx-entry-load-bndcfgs
  vmx-entry-load-cet vmx-entry-load-efer vmx-entry-load-fred
  vmx-entry-load-pat vmx-entry-load-perf-global-ctrl vmx-entry-load-pkrs
  vmx-entry-load-rtit-ctl vmx-entry-noload-debugctl vmx-ept vmx-ept-1gb
  vmx-ept-2mb vmx-ept-advanced-exitinfo vmx-ept-execonly vmx-eptad
  vmx-eptp-switching vmx-exit-ack-intr vmx-exit-clear-bndcfgs
  vmx-exit-clear-rtit-ctl vmx-exit-load-efer vmx-exit-load-pat
  vmx-exit-load-perf-global-ctrl vmx-exit-load-pkrs
  vmx-exit-nosave-debugctl vmx-exit-save-cet vmx-exit-save-efer
  vmx-exit-save-pat vmx-exit-save-preemption-timer vmx-exit-secondary-ctls
  vmx-flexpriority vmx-hlt-exit vmx-ins-outs vmx-intr-exit vmx-invept
  vmx-invept-all-context vmx-invept-single-context
  vmx-invept-single-context vmx-invept-single-context-noglobals
  vmx-invlpg-exit vmx-invpcid-exit vmx-invvpid vmx-invvpid-all-context
  vmx-invvpid-single-addr vmx-io-bitmap vmx-io-exit vmx-mbec
  vmx-monitor-exit vmx-movdr-exit vmx-msr-bitmap vmx-mtf vmx-mwait-exit
  vmx-nested-exception vmx-nmi-exit vmx-page-walk-4 vmx-page-walk-5
  vmx-pause-exit vmx-ple vmx-pml vmx-posted-intr vmx-preemption-timer
  vmx-rdpmc-exit vmx-rdrand-exit vmx-rdseed-exit vmx-rdtsc-exit
  vmx-rdtscp-exit vmx-secondary-ctls vmx-shadow-vmcs vmx-store-lma
  vmx-true-ctls vmx-tsc-offset vmx-tsc-scaling vmx-unrestricted-guest
  vmx-vintr-pending vmx-vmfunc vmx-vmwrite-vmexit-fields vmx-vnmi
  vmx-vnmi-pending vmx-vpid vmx-wbinvd-exit vmx-xsaves vmx-zero-len-inject
  vnmi vpclmulqdq waitpkg wbnoinvd wdt wrmsrns x2apic xcrypt xcrypt-en xfd
  xgetbv1 xop xsave xsavec xsaveerptr xsaveopt xsaves xstore xstore-en xtpr
  zero-fcs-fds

This is obviously incorrect. even if QEMU will boot while ignoring flag. On the other hand, Hyper-V enlightenment flags are not present in this list at all, no matter what CPU QEMU runs on (tried few Intel and EPYC CPUs), which causes PVE to show them as incompatible:


The change should likely be reverted or minimally, Hyper-V enlightenment should be excluded as QEMU does not seem to treat them in same way as other flags.

 

[a.bied-charreton]

Hey!

To your first point: the new feature uses the -cpu help output as a base and populates the per-node support field with the flags that the host CPUs actually advertise [0]. Those are retrieved by spawning a temporary VM with the CPU type "host" and querying its capabilities via QMP [1]. The base list is intentionally the cross-vendor catalog of every flag QEMU knows about, which is why you will see AMD-only entries like 3dnow on an Intel host. The "Supported-on"-column is what is meant to filter those out.

As for the hv-* flags, QEMU advertises those to the guest through a different CPUID range rather than the standard feature words [2]. Since query-cpu-model-expansion only enumerates the standard feature words of the requested CPU model, it never returns any hv-* keys on any host so we end up reporting them as unsupported. Host support for them is exposed through a different capability check (qom-list-properties) which the current detection path does not consume.

I don't think a revert is necessary, but the hv-* handling does need fixing. The availability of those flags is a property of QEMU+KVM rather than of the host CPU. I think the cleanest fix is to special-case the hv-* set the same way nested-virt already is in query_available_cpu_flags [0] i.e., mark them as supported on every KVM-capable x86_64 node.

[0] https://git.proxmox.com/?p=qemu-ser...6785065b3f766f15f6f151af8ec27ec8bb5b07ab#l252
[1] https://git.proxmox.com/?p=qemu-ser...785065b3f766f15f6f151af8ec27ec8bb5b07ab#l2941
[2] https://www.qemu.org/docs/master/system/i386/hyperv.html

 

[eider]

Thanks for clarification. For reference though, regarding first point - I do see amd-ssbd with "Supported on" listing nodes that are running Intel CPUs. Is this expected?

 

[a.bied-charreton]

KVM is our source of truth for CPU flag support, and it reports this flag (as well as others) regardless of the underlying hardware. I agree the vendor prefix in the name makes this confusing.

However, this is correct in the sense that QEMU will accept the flag and the VM will start successfully, it is just a no-op on Intel hardware. The "Supported On" field shows the list of nodes that can successfully start a VM using a given flag, which is useful for understanding which nodes a VM may be migrated to.

Reporting this flag as unsupported, while correct from a hardware point of view, would actually be misleading because it would imply that migrating a VM using this flag to that node would fail, even though it would not.

 

[leesteken]

My host CPU is much like EPYC-MIlan but does not support pcid. It would be nice if the Proxmox GUI could show the flags supported by the chosen Base Model (but not by the host CPU) so it can be turned off. The current work-around is to add -pcid manually to the configuration.

 

[abufrejoval]

I guess you get thrown off by the "amd-ssbd" and the "amd-stibp" flags on Intel CPUs.

I just had a look at those flags ony my mixed CPU infra, which includes some Goldmont Intel Atoms as well as Zen3/4/5 hosts and was somewhat thrown off by Intel CPUs supporting "AMD" micro-code enhancements for Spectre/Meltdown bypasses.

I'd say it's mostly like 64-bit x86 being called AMD64, whoever documented a capacity (or resilience) first, got the brand name in the moniker.

Those flags aren't designed to reflect actual capabilities, but to manage shared baselines to enable live-migration (or operation) of VMs whith a minimum required set of vulnerability protection.

So if AMD documented that they aren't exposed to a given type of vulnerability, that might have gotten an AMD moniker to attest it's "resilient" against such attack types. And once Intel CPUs had microcode fixes implemented or got re-designed to avoid that from scratch, they might also earn that tag or badge, that might now have that pesky "amd" in its name.

The aim is twofold: 1) allow VMs that require a certain protection to only run on machines that support them 2) to hide "higher" resilience levels from a VM that initially ran on a better protected or featured host, because applications which inquire exact hardware capabilities and adjust to them, would then potentially fail when migrated or revived on a CPU that only implements a lower subset.

It's just the mess we live in, it can only get more complicated as ISA extensions and vulnerabilities keep exploding, perhaps even worse on ARM and RISC-V.

 

[eider]

I guess you get thrown off by the "amd-ssbd" and the "amd-stibp" flags on Intel CPUs.

Not only. The description of the flag provided by PVE itself is very explicit in saying it applies only to AMD CPUs:

Improves Spectre mitigation performance with AMD CPUs, best used with 'virt-ssbd'.

Incidentally, virt-ssbd is properly marked as not supported on Intel CPUs, which only add to the whole mess.

Also looking at QEMU own documentation, amd-ssbd is listed under flags for AMD hosts, while Intel section just has ssbd

Reporting this flag as unsupported, while correct from a hardware point of view, would actually be misleading because it would imply that migrating a VM using this flag to that node would fail, even though it would not.

Possibly it could be folded in similarly to nested-virt, though that creates it's own can of worms on hardware that legitimately only supports virt-ssbd but not amd-ssbd (older Opterons?)