[SOLVED] Mixed cluster with Intel+Amd: any chance for live migration? [Solved by: not really possible]

[UdoB]

Hello everybody,

while there are already several threads regarding Intel & Amd in a mixed cluster I am not sure about the current state. Most threads I've read drift away in one or another direction. So please forgive me if you feel this has already been discussed enough...

My cluster is currently mixing "Intel Xeon Gold 5218" and "AMD EPYC 7443" and I am on 5.19.7-2 kernel with qemu-server 7.2-4 and pve-qemu-kvm 7.0.0-3. All nodes have the latest software from the Enterprise repository.

• VMs with kvm64 crash immediately or within 20 minutes(!) after live migration, so we just avoid this
• now I run tests with qemu64 and a Test-VM has successfully survived half a dozen live migrations

Searching for "qemu64 vs kvm64" gave me no helpful advice. https://qemu.readthedocs.io/en/latest/system/qemu-cpu-models.html simply recommends NOT to use both of them. (qemu64 for security reasons, kvm64 mainly for performance.)

So the question is: do I just have to live without live migration or does anyone know another trick to allow stable live migrations between my architectures?

Best regards

 

[apoc]

So the question is: do I just have to live without live migration or does anyone know another trick to allow stable live migrations between my architectures?

Technically you can use live migration between and and Intel.
What it does though: it disables everything that leads to performance. This is exactly what you are dong with qemu64 architecture. You are emulating your CPU throwing all performance out of the window.
I have not made any tests but I guess you waste 90% of your CPU performance for your flexibility.
Even kvm64 is suboptimal in my opinion. You should use "host CPU" setting to get best performance.

This topic is not solvable as we talk about multiple differences in the CPU and its design as well as instruction sets.

Same if you would want to mount the engine from a Tesla into a classic car with gas-engine. It would not work without sacrificing a lot of benefits.

 

[UdoB]

...throwing all performance out of the window.
I have not made any tests but I guess you waste 90% of your CPU performance for your flexibility.

Thanks for the answer. That may absolutely be true. However performance is (currently) not my problem and I did not run performance tests in this regard (yet). Maximum CPU-usage for the most stressed node today was 6 times 10 minutes with 40% and below 5% the rest of the day. So yes, a factor of 10 probably would make me unhappy.

Currently I am just poking around to find the specific optimum in usefulness / usability / flexibility. Living with half a cluster of Amd and the other half with Intel is possible for sure - it just requires more carefulness as expected...

Best regards

 

[apoc]

Another Option would be to build two discrete clusters which can migrate within online and cross-cluster migrate offline (if you have enough nodes).
Might in the end be a much better solution from a performance and usability perspective.

 

[UdoB]

Another Option would be to build two discrete clusters which can migrate within online and cross-cluster migrate offline (if you have enough nodes).

Yeah, thanks for the idea. We are too small for this. Currently I have only two AMD and one Intel and a second one will follow the next days. (And with four nodes I will add a Qdevice.) That's it. Additional AMDs will be added in the future, when the Intel nodes will get replaced. But the future is... only available in the future.

Best regards

 

[Matthias.]

You can find some threads on the forum where users discuss their experience and some performance tips, e.g. https://forum.proxmox.com/threads/l...-intel-xeon-and-amd-epyc2-linux-guests.68663/

Test and benchmark and decide for yourself if the result is worth it : )

 

[UdoB]

Thanks @Matthias. Actually I did "watch" that thread already
The list of CPU flags is interesting and I need to reinvestgiate...

 

[Neobin]

Currently I have only two AMD and one Intel and a second one will follow the next days.

Especially regarding the topic/problem, would it not have been better to get a third AMD, instead of the second Intel and look forward to replace the last Intel in the (near) future too?!

I (blindly) assume (I do not know, if it could potentially be (more) mitigated on the software-side) the circumstances with live-migration between mixed architectures will forever be there (more or less).

But I am sure, there is a reason for the choice/decision. I would be interested in the reason, out of curiosity and for my knowledge.

 

[UdoB]

Especially regarding the topic/problem, would it not have been better to get a third AMD

Yes, definitely! But reality has some strange constraints regarding the availability of hardware listed on the wishlist ;-)

Not really secret information: we are migrating from that well-known-big-standard-solution to PVE. And we are switching from Intel to Amd at the same time. (If not now when?) For starting this I was able to buy two EPYCs (Dell). But some other Intel hardware (from the then obsolete other cluster) has to "recycled" and will be used for another year or two.

This is a single Institute at a University in Germany. Money is a limited resource...

Best regards

 

[UdoB]

Marked "Solved" - although it is more a "it is what it is"...

 

[Neobin]

Yes, definitely! But reality has some strange constraints regarding the availability of hardware listed on the wishlist ;-)

Not really secret information: we are migrating from that well-known-big-standard-solution to PVE. And we are switching from Intel to Amd at the same time. (If not now when?) For starting this I was able to buy two EPYCs (Dell). But some other Intel hardware (from the then obsolete other cluster) has to "recycled" and will be used for another year or two.

This is a single Institute at a University in Germany. Money is a limited resource...

Best regards

Ah, okay. Thank you very much for the explanation/insights.