Proxmox on aarch64 (arm64)

[Johannes S]

Ugh. That sounds absolutely horrible for troubleshooting/debugging any problems that occur.

On the other hand together with it's immutable state it should make tampering though bad actors a lot harder.

The same approach is used by kubernetes distribution talos and less extremely TrueNAS ( the latter still has ssh though, talos has not, both expect users to use the api or tools/uis based on the UI) so I can see the appeal from a devops perspective. Personally I prefer the flexibility of ProxmoxVE. Like always: Pick your poison

 

[justinclift]

less extremely TrueNAS

Um, as you mention TrueNAS does have ssh, so it's very much not the same as these things that can't be accessed.

Also, am kind of curious why you'd say not having ssh is a feature for DevOps? How are you expecting DevOps staff to diagnose problems if they don't even have shell access? Maybe these things have terminal/console/tty access instead, which would do effectively the same thing in a pinch...?

 

[Johannes S]

Um, as you mention TrueNAS does have ssh, so it's very much not the same as these things that can't be accessed.

Yes, but the TrueNAS documentation clearly states that you are expected not to alter the system but use the API, the WebUI or the TrueNAS specific cli tools. So although you can use ssh it's less powerful than on a classical Unix box (since you can't do much on the system). Whether this is a good thing or not, depends on your point of view
My point concerning devops is that if you happen to be a former developer who somehow ended up in devops it's very appealing to do everyting with an API and a debug channel (maybe prometheus metrics or an endpoint for syslogs), because that is what you are used to. For sysadmins (which I am and I suspect you too) who are used to do everything via ssh the lack of ssh is an antifeature

 

[justinclift]

Heh Heh Heh. The good developers i know (and work with) very much understand how to use ssh and a shell effectively. Doing everything over an API (etc) is just such a huge waste of time/effort for debugging system problems.

But yeah, I could see how juniors who don't know the fundamentals might not understand the value.

 

[ambo]

That's all there is for Proxmox here ... homelabs .... no income to gain so not worth it commercially. I'd also love it but there is not enough special software for aarch64 that no other architecture has - besides the Mac eco system. It's still a niche market. Powerful aarch64 hardware is not cheaper than x86-64, so only one disadvantage after another.

There is a major factor which I think has been ignored: The US has placed sanctions on China and many Chinese technology companies. This blocks access to many American technologies - such as Intel and AMD CPUs - for these companies. These tech companies have pivoted - and are now putting all of their efforts behind ARM based hardware and software which is compatible with those platforms.

This hardware isn't yet showing up in massive volumes in places like eBay for home labbers - but enterprise grade hardware certainly exists. Systems like TaiShan are being produced in the thousands (perhaps millions). These are being sold both in Asia - and in many other markets around the world as well. A large percentage (potentially the majority) of services in these regions already runs on top of ARM - whether we realise it or not.

Much of the adoption of ARM is happening "behind the curtain". Hyperscalers (like AWS) are adopting ARM CPUs due to the power efficiencies. Even where end customer adoption of ARM based compute instances may be slow - for services like hosted databases its easy to swap out the underlying hardware without anyone noticing.

 

[Johannes S]

Much of the adoption of ARM is happening "behind the curtain". Hyperscalers (like AWS) are adopting ARM CPUs due to the power efficiencies. Even where end customer adoption of ARM based compute instances may be slow - for services like hosted databases its easy to swap out the underlying hardware without anyone noticing.

But hyperscalers won't use ProxmoxVE but their own KVM/qemu-based virtualization stack.

I agree with your other points though

 

[jlauro]

But hyperscalers won't use ProxmoxVE but their own KVM/qemu-based virtualization stack.

I agree with your other points though

There are some smaller cloud companies that do use proxmox, and probably more that use openshift then proxmox, but I know of at least a couple in the thousands of nodes that publicly stated they are based on it. If they would consider using ARM or not is another question. Granted out of literally hundreds of cloud providers, the top 4 have 60% of the market so the remaining ones are relatively small...

A hybrid of their own stack on top of proxmox is a lot easier/quicker than starting from simply KVM/qemu alone.

 

[justinclift]

This is interesting. Looks like VMware is testing the waters with ARM support: https://www.theregister.com/virtual...ly-debuts-arm-hypervisor-tech-preview/5242293

 

[risner]

VMWare on Arm has been a thing since 2020. At that time, I believe it was RPI only.

https://www.storagereview.com/news/vmware-esxi-arm-fling-announced

I'm kinda puzzled why they would continue to create it, but maybe they view it as a revenue source (if they are the only/major ARM virtualization provider).

It's possible they also ship a Mac version, as they support all the Intel macs up to 7.x releases.

 

[Johannes S]

The official port is there, no support for raspberry though: https://forum.proxmox.com/threads/p...nt-now-available-for-64-bit-arm-arm64.185526/

 

[jlauro]

Interesting. Might have to consider a different platform next cluster purchase. Can it run on a Mac and support Mac guests? or is it only good for Superchip server systems.

 

[Johannes S]

Interesting. Might have to consider a different platform next cluster purchase. Can it run on a Mac and support Mac guests? or is it only good for Superchip server systems.

As far I know the development work was mostly done for Nvidias arm-based AI platforms as part of the new partnership between Proxmox Server Solutions and nvidia, other arm platforms are supported on a "best effort"-base. They need uefi boot support though which rule out most sbcs like the pi. No idea how it's on Apple or miniforums arm workstation

 

[alexskysilk]

Might have to consider a different platform next cluster purchase.

there are very few options for this. and because its all low volume, they are not at all cost or performance competitive with comparable x86.

Can it run on a Mac and support Mac guests?

No. and even if it could run, it wouldnt support mac guests, support MLX either directly or indirectly. see https://asahilinux.org/

 

[jlauro]

there are very few options for this. and because its all low volume, they are not at all cost or performance competitive with comparable x86.


No. and even if it could run, it wouldnt support mac guests, support MLX either directly or indirectly. see https://asahilinux.org/

You are probably right about pure ARM without GPUs, but looking at the options, most of the prices don't look that bad comparing to x86, especially when the largest portion of the price is the GPUs and memory and the performance of the CPUs don't matter much. It's likely our next cluster will have enterprise GPUs for AI.

Based on what I seen and heard it could run on a Mac. Proxmox implies it when they say you can install Proxmox under Trixie and you can get Trixie installed on native hardware on a Mac. That said, I agree unlikely to get guest Mac systems to run under Proxmox (and without being able to run Mac OS under it, little reason to run Proxmox on a Mac).

 

[alexskysilk]

Proxmox implies it when they say you can install Proxmox under Trixie and you can get Trixie installed on native hardware on a Mac.

While it is possible to install Trixie on Mac hardware, it is dependent on Asahi linux patches. I doubt the pve Kernel would even work. If this is something of specific interest to you, please try it and let us know

this would be enlightening reading: https://wiki.debian.org/InstallingDebianOn/Apple/M1

most of the prices don't look that bad comparing to x86

Thats not a good sales pitch. "not that bad" has to overcome different ABI considerations, kernel optimizations, poor IPC, poor single thread performance, and poor ppw. Even if it was the same price or slightly cheaper then an equivalent x86 server I still wouldnt buy it.

 

[jlauro]

While it is possible to install Trixie on Mac hardware, it is dependent on Asahi linux patches. I doubt the pve Kernel would even work. If this is something of specific interest to you, please try it and let us know

this would be enlightening reading: https://wiki.debian.org/InstallingDebianOn/Apple/M1


Thats not a good sales pitch. "not that bad" has to overcome different ABI considerations, kernel optimizations, poor IPC, poor single thread performance, and poor ppw. Even if it was the same price or slightly cheaper then an equivalent x86 server I still wouldnt buy it.

I don't have a lot of time or mac hardware, but will put it on my backlog to try... probably not for a month or two. I'm sure someone will get to it before me.

I assume the distros and compiler writers have done enough kernel optimizations, as many companies have put a lot of money into supporting ARM. That said, everything I have read is that excellent ppw is one of the main reasons people pick it over Intel or even AMD so surprised you say poor ppw when it's generally has the best ppw in the server market, the second being core density, so it makes up with quantity over single thread quality.

 

[alexskysilk]

You might want to cite your sources. and see what they were comparing.

 

[jlauro]

You might want to cite your sources. and see what they were comparing.

It's pretty well known, I can't find anything that states otherwise that isn't over a decade old. Here is couple, but google is pretty easy to find more.
https://www.tomshardware.com/pc-com...2029-x86-and-risc-v-on-the-outside-looking-in
Here is a couple more:
https://unihost.com/blog/arm-vs-x86-servers-architecture-guide/

https://www.linkedin.com/pulse/data-center-cpus-2025-architecture-availability-strategy-xyyae/


What's your sources?

 

[alexskysilk]

It's pretty well known, I can't find anything that states otherwise

Absence of evidence isnt evidence of absence.

but google is pretty easy to find more.

Searching for evidence after making a claim is backwards.

your google searches arent even informative- the toms hardware is breathless marketing without ANY substance. the Unihost article discusses theory, not actual metal. and I wont even dignify an article on linkedin with a comment; if you dont understand why I dont think anything I say would be convincing to you. There is no question that in the right circumstance and workload ARM makes perfect sense, but only at scale; Amazon, Microsoft, and Google all have enough for a bespoke design to make sense- but the only part commercially available is Ampere Altra. You dont need any think pieces, you have access to datasheets:



Even if I was to compare it to a DDR4 generation part, say an Epyc 7713P (64C/128T) which has a 225W TDP and will outperform the Q80 in virtually any load and the MI128-30 in most loads. For any load that requires single thread performance, that generation ARM is a non starter at all.

But thats not the worst of it. Once adding 8 channels of DDR4 ram, 128 PCIe lane retimers, NICS, HBAs, etc, the power difference is nearly meaningless. the Epyc part will be cheaper to obtain and run standard loads, would be able to participate in clusters with other hardware, will have standardized kernel and driver stack, etc etc etc.

If I was to compare it to SOTA, either Bergamo/Turin Dense or Intel Xeon 6 Sierra Forest would run circles around those ARM designs. Your original "its pretty well known" assertion suffers time decay.