Recommended VM CPU flags for mixed-vendor cluster?

Russell Ault

Member
Nov 28, 2020
6
0
6
36
Following on from this forum thread and this bug report, I was wondering what the current recommendation is for Linux VM CPU flags (or other settings) to allow migrating VMs between AMD- and Intel-based hosts? I couldn't find anything in the documentation, the most recent advice on that forum thread is approaching two years old, and the most recent update on that bug report was over a year ago.

Does anyone have more up-to-date information?

Thanks!

-Russ
 
you should keep the default kvm64 cpu model, but even with that, it's not 100% safe, sometime it doesn't work.
In my experience (and that of others, according to the links I mentioned above) kvm64 along with the default flags guarantees failure (well, technically, "migration success", followed by a total hard-lock of the VM at some point afterwards).

The blog post I referenced suggested the following for CPU flags for Linux VMs:
Code:
kvm64,+ssse3,+sse4.1,+sse4.2,+x2apic,+aes,+sep,+ibpb,+movbe,+lahf_lm,+virt-ssbd,+kvm_pv_eoi

Does that still seem reasonable? Conversely, can someone point me towards a document that lists and explains all the VM CPU flags currently available in PVE?

Thanks!

-Russ
 
I think it's depend also of the amd cpu model

here the cpu flags on my diffents amd generation


AMD Opteron(tm) Processor 6172

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save pausefilter


AMD Opteron(tm) Processor 6378

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold


AMD EPYC 7542 32-Core Processor

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

and intel

Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit

Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips : 6802.03


The predefined qemu cpu models flags are available in qemu src code
https://github.com/qemu/qemu/blob/cb5ed407a1ddadf788fd373fed41c87c9e81e5b0/target/i386/cpu.c


+ssse3,+sse4.1,+sse4.2, -> (memory optimisation, optionnal, can't accelerate applications if they are compiled to support it)
+x2apic, (intel + amd epyc only)
+aes, -> hardware crypto (optionnal, to speedup ssl for example)
+sep, (avaible on intel/amd)
+ibpb, (spectre/meldown protection, amd only) -> optionnal
+movbe, (avaiable on intel, and amd epyc only)
+lahf_lm, (Load AH from Flags (LAHF) and Store AH into Flags (SAHF) in long mode) -> avaiable on intel/amd

+virt-ssbd, (spectre meldown like protection, protection, it's for amd only. intel flag is +ssbd. don't known how it can work...)
+kvm_pv_eoi : paravirtual end of input (EOI) events. (it's kvm specific, for acceleration)

don't remember about the other.

I never have tested between intel and epyc, but on older amd (not epyc), the migration was sometime suceffull, but I had kernel panic some hours later.
I think it could be great to find the correct flags, and add a new model in proxmox directly.
I'm not sure about spectre/meltdown protections, how it's possible to have them in both amd/intel.

Maybe try with basic like (+x2apic, +sep, +movbe,+lahf_lm, +kvm_pv_eoi) and see if it's working. (and make them running for 24h after migration, to see if they are not crashing later)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!