[SOLVED] Mixed cluster with Intel+Amd: any chance for live migration? [Solved by: not really possible]

UdoB

Distinguished Member
Nov 1, 2016
1,811
687
183
Germany
Hello everybody,

while there are already several threads regarding Intel & Amd in a mixed cluster I am not sure about the current state. Most threads I've read drift away in one or another direction. So please forgive me if you feel this has already been discussed enough...

My cluster is currently mixing "Intel Xeon Gold 5218" and "AMD EPYC 7443" and I am on 5.19.7-2 kernel with qemu-server 7.2-4 and pve-qemu-kvm 7.0.0-3. All nodes have the latest software from the Enterprise repository.
  • VMs with kvm64 crash immediately or within 20 minutes(!) after live migration, so we just avoid this
  • now I run tests with qemu64 and a Test-VM has successfully survived half a dozen live migrations
Searching for "qemu64 vs kvm64" gave me no helpful advice. https://qemu.readthedocs.io/en/latest/system/qemu-cpu-models.html simply recommends NOT to use both of them. (qemu64 for security reasons, kvm64 mainly for performance.)

So the question is: do I just have to live without live migration or does anyone know another trick to allow stable live migrations between my architectures?

Best regards
 
So the question is: do I just have to live without live migration or does anyone know another trick to allow stable live migrations between my architectures?
Technically you can use live migration between and and Intel.
What it does though: it disables everything that leads to performance. This is exactly what you are dong with qemu64 architecture. You are emulating your CPU throwing all performance out of the window.
I have not made any tests but I guess you waste 90% of your CPU performance for your flexibility.
Even kvm64 is suboptimal in my opinion. You should use "host CPU" setting to get best performance.

This topic is not solvable as we talk about multiple differences in the CPU and its design as well as instruction sets.

Same if you would want to mount the engine from a Tesla into a classic car with gas-engine. It would not work without sacrificing a lot of benefits.
 
  • Like
Reactions: tjk
...throwing all performance out of the window.
I have not made any tests but I guess you waste 90% of your CPU performance for your flexibility.
Thanks for the answer. That may absolutely be true. However performance is (currently) not my problem and I did not run performance tests in this regard (yet). Maximum CPU-usage for the most stressed node today was 6 times 10 minutes with 40% and below 5% the rest of the day. So yes, a factor of 10 probably would make me unhappy.

Currently I am just poking around to find the specific optimum in usefulness / usability / flexibility. Living with half a cluster of Amd and the other half with Intel is possible for sure - it just requires more carefulness as expected...

Best regards
 
Another Option would be to build two discrete clusters which can migrate within online and cross-cluster migrate offline (if you have enough nodes).
Might in the end be a much better solution from a performance and usability perspective.
 
Another Option would be to build two discrete clusters which can migrate within online and cross-cluster migrate offline (if you have enough nodes).
Yeah, thanks for the idea. We are too small for this. Currently I have only two AMD and one Intel and a second one will follow the next days. (And with four nodes I will add a Qdevice.) That's it. Additional AMDs will be added in the future, when the Intel nodes will get replaced. But the future is... only available in the future.

Best regards
 
Currently I have only two AMD and one Intel and a second one will follow the next days.

Especially regarding the topic/problem, would it not have been better to get a third AMD, instead of the second Intel and look forward to replace the last Intel in the (near) future too?!

I (blindly) assume (I do not know, if it could potentially be (more) mitigated on the software-side) the circumstances with live-migration between mixed architectures will forever be there (more or less).

But I am sure, there is a reason for the choice/decision. I would be interested in the reason, out of curiosity and for my knowledge. :)
 
Especially regarding the topic/problem, would it not have been better to get a third AMD
Yes, definitely! But reality has some strange constraints regarding the availability of hardware listed on the wishlist ;-)

Not really secret information: we are migrating from that well-known-big-standard-solution to PVE. And we are switching from Intel to Amd at the same time. (If not now when?) For starting this I was able to buy two EPYCs (Dell). But some other Intel hardware (from the then obsolete other cluster) has to "recycled" and will be used for another year or two.

This is a single Institute at a University in Germany. Money is a limited resource...

Best regards
 
  • Like
Reactions: Neobin
Yes, definitely! But reality has some strange constraints regarding the availability of hardware listed on the wishlist ;-)

Not really secret information: we are migrating from that well-known-big-standard-solution to PVE. And we are switching from Intel to Amd at the same time. (If not now when?) For starting this I was able to buy two EPYCs (Dell). But some other Intel hardware (from the then obsolete other cluster) has to "recycled" and will be used for another year or two.

This is a single Institute at a University in Germany. Money is a limited resource...

Best regards

Ah, okay. Thank you very much for the explanation/insights. :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!