Live migration, CPU types and performance

[Jackobli]

Hello

We are running PVE 8.1.3 on HPE DL 325/365 Gen 10 servers with different CPUs.
Usually we only use 6 DL 325 with AMD EPYC Rome (7502P and 7502P).
A while ago we added a seventh server which has a Milan 7513 CPU (two sockets).
While doing live migrations for lifecycle (Patching) purpose we had some machines crashing. We found out that migration from Rome to Milan seems to be ok, but migration back from Milan to Rome made some of the machines unresponsive or even crash/reboot.
Usually we run vms with qemu cpu type «default», which means IMHO kvm64. For performance reasons (back in PVE 7) some of the vm had other cpu types.
So we went and did build test machines (2 vCPU, Alma Linux) and migrated them forth and back.
Sorry for using a picture to explain, it was easiest.

We wonder, why are there so many different EPYC cpu types. And especially, why Rome seems not to run on a Rome CPU. Nero seems to have burnt it down

Also, our performance test (sysbench) shows average performance on (mostly idle) hosts
- Best performance on a 7502P (Rome) with QEMU cpu type = EPYC IBPB (12% more events per second as with host (lowest))
- Best performance on a 7513 (Milan) with QEMU cpu type = EPYC (18% more events per second as with kvm64 (lowest))

We found different sites with different explanations or reasons and would like to know, what would be best.

Regards, Urs

 

[itNGO]

Hi,
live migration works best with identical CPUs and setting CPU-Type HOST.
When you have different Vendors, INTEL and AMD. Live Migration can work in some specific scenarios and settings.

Between ROME and MILAN, live migration works upwords 100%. This was already confirmed by your tests. Down-Migration to an older CPU can work, but sometimes internal timings in the CPUs differ to much. This can cause "hickups" even when setting an "emulated" CPU-Type like "EPYC".

EPYC Milan has several internal changes compared to ROME and this can cause "problems" with Live Migration.

The basic idea is, to set the CPU-Type to the "lowest" capable CPU you have in cluster and want to use for live migration. But even this is not a 100% guarantee. It just increases the chance, that it will work....

 

[Jackobli]

Hi,
live migration works best with identical CPUs and setting CPU-Type HOST.
When you have different Vendors, INTEL and AMD. Live Migration can work in some specific scenarios and settings.

Between ROME and MILAN, live migration works upwords 100%. This was already confirmed by your tests. Down-Migration to an older CPU can work, but sometimes internal timings in the CPUs differ to much. This can cause "hickups" even when setting an "emulated" CPU-Type like "EPYC".

EPYC Milan has several internal changes compared to ROME and this can cause "problems" with Live Migration.

The basic idea is, to set the CPU-Type to the "lowest" capable CPU you have in cluster and want to use for live migration. But even this is not a 100% guarantee. It just increases the chance, that it will work....

Could…, should…, perhaps… are not terms I really like in IT.
Have been running several hypvisors and as you write, usually the hypervisor has the capability to mask cpu features, so a migration will work and if not, it doesn't allow a migration. For sure I would not try to mix Intel and AMD, not even older Athlons with newer EPYCs.
Earlier we were stunned that a migration was initiated but a large amount of vms silently crashed.
I also don't understand why there are so many «Rome» types, but are not booting on a Rome.

 

[spirit]

about xsaves, it has been removed with last amd microcode update, but the qemu model without xsaves is not yet added in proxmox.

EPYC-Rome-v4


That mean than EPYC-Rome-v2 is not working on your epyc rome if you have done last microcode update


as workaround, you can create a custom model

/etc/pve/virtual-guest/cpu-models.conf

cpu-model: EPYC-Rome-noxsaves
    flags -xsaves
    reported-model EPYC-Rome