ARM Hardware

batijuank

Member
Nov 16, 2017
45
0
11
31
Can Proxmox VE be installed on ARM server? If so, those anyone has done it yet? And finally, I'm currently working in a small factory (less than 100 users), we want to setup a High Aviability Cluster with 3 nodes, since our workload per user is not high, we thought of using ARM servers, is this possible?
 
Last edited:

guletz

Renowned Member
Apr 19, 2017
1,409
212
68
Brasov, Romania
Hi,

As I remember you can not use proxmox on ARM, but if your ARM cpu have hw virtualization you can create a VM and then you can install proxmox in this VM. But I think that this is not a very nice setup. I test for myself some years ago on a ARM (odrod c2)and the performance was not I have expected; )
 
  • Like
Reactions: batijuank

batijuank

Member
Nov 16, 2017
45
0
11
31
That's a shame, I really like the idea of having ARM CPU servers with Proxmox, as they are cheaper and consume less power, for small business with low workloads, this is something to worth having.

Anyway, I would like this info you gave to be confirmed by Proxmox itself.
 

wbumiller

Proxmox Staff Member
Staff member
Jun 23, 2015
673
98
48
There's a work-in-progress with some patches to get things going on the devel list which is runnable, but there's no official support or release yet (and therefore incompatibilities to be expected between versions). ARM hardware can be a bit annoying to support in a "generic" way, kernel-wise, and unfortunately many of the smaller platforms which I'd expect people to be testing around with (and which would therefore be nice to be working out of the box) either don't provider kernel sources in a usable way, or with no clear upstream base to figure out what they're doing, or not at all, and often don't work with mainline kernels, and/or take so long to get things upstream that they're simply not of interest anymore by then...
 

Jianwen Zhu

New Member
Nov 7, 2018
1
1
1
41
I suggest to only support ARM servers and the available server platforms in this case are really limited. We have access to some
server boxes and can volunteer to test.

Re whether it is economically justified to
support — I think it is about preparing for
the future. It is rather chick-and-egg problem:
without software like proxmox in the ecosystem, ARM will never take off;
And without ARM server having some
momentum, software is never justified
to develop. I do think for the same
economic reason VMWare won’t
get into ARM, but that’s where Proxmox
may carve some niche. My 2 cents.

There's a work-in-progress with some patches to get things going on the devel list which is runnable, but there's no official support or release yet (and therefore incompatibilities to be expected between versions). ARM hardware can be a bit annoying to support in a "generic" way, kernel-wise, and unfortunately many of the smaller platforms which I'd expect people to be testing around with (and which would therefore be nice to be working out of the box) either don't provider kernel sources in a usable way, or with no clear upstream base to figure out what they're doing, or not at all, and often don't work with mainline kernels, and/or take so long to get things upstream that they're simply not of interest anymore by then...
 
  • Like
Reactions: LEE THOONG CHING
Nov 13, 2018
7
0
6
40
I suggest to only support ARM servers and the available server platforms in this case are really limited. We have access to some
server boxes and can volunteer to test.

Re whether it is economically justified to
support — I think it is about preparing for
the future. It is rather chick-and-egg problem:
without software like proxmox in the ecosystem, ARM will never take off;
And without ARM server having some
momentum, software is never justified
to develop. I do think for the same
economic reason VMWare won’t
get into ARM, but that’s where Proxmox
may carve some niche. My 2 cents.
Which model of ARM server are you working on now ? ThunderX2 ? We really want to explore the possibility of going N cores with many motherboards with many Gbps to run LXC as many as we can.
 

tonyrich

New Member
Feb 17, 2021
1
0
1
41
Now that Debian is available for Raspberry Pi and the hardware is available in a 64-bit version with 8Gb RAM, is it possible to provide an arm64 repository so we can add Proxmox to an existing Debian install on this platform?
I can imagine some sweet little setups, for more than one use case. I have a ton of single service LXCs that would happily run on a Pi, for example, so something like four 4b/8g Raspis, with POE hats and a TP-Link TL-SG1005P (or something similar) would make a nice little ultra portable cluster setup, with 16 cores and 32 gb ram. Also.. Hello, ESXi-Arm Fling.. :)
 

JVini0166

New Member
Feb 19, 2021
4
0
1
18
Hey Proxmox, let's update! The most powerful computer in the world it's a ARM based one with many Petabytes, and also there are many ARM boards that are cheap, has less consumption and are more efficient than x86, VMware ESXi is now compatible in ARM boards, let's do a Proxmox ARM version too?! I love Proxmox and I would like to have in my ARM Servers
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
14,819
695
133
@JVini0166 , please send me a link where those "powerful arm server" can be purchased in Europe.
 

JVini0166

New Member
Feb 19, 2021
4
0
1
18
https://www.arm.com/blogs/blueprint...heart of Fugaku,based processors in the world.

Here is the most powerful computer in the world, and based in ARM (they will not invest in something that costs more right? In cases of energy consumption and more)

If you want to purchase some simple one that are many companies that you can order by specs, there isn't link you need to order it, but if you want something that has a capacity of a simple Xeon for 50$ or less you can use Raspberry Pi, mine working with 8 VMs usign KVM as Cloud Servers :)
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
3,678
754
163
South Tyrol/Italy
shop.maurer-it.com
https://www.arm.com/blogs/blueprint...heart of Fugaku,based processors in the world.

Here is the most powerful computer in the world, and based in ARM (they will not invest in something that costs more right? In cases of energy consumption and more)
Super computers just cannot be compared to server HW, they for very specialiced problems which just need lots of floating point operations. Completely useless for virtualization and most server applications. That's like posting a blog post for a GPU hash rig, just because its "powerful" in someway (doing lots of hashes) does not makes it a good target for Proxmox VE or the like.

If you want to purchase some simple one that are many companies that you can order by specs, there isn't link you need to order it
Where order it?

has a capacity of a simple Xeon for 50$ or less you can use Raspberry Pi, mine working with 8 VMs usign KVM as Cloud Servers :)
A Raspberry just really does not compare with a modern XEON processor, I worked with both quite a bit.
 
Last edited:

JVini0166

New Member
Feb 19, 2021
4
0
1
18
Super computers just cannot be compared to server HW, they for very specialiced problems which just need lots of floating point operations. Completely useless for virtualization and most server applications. That's like posting a blog post for a GPU hash rig, just because its "powerful" in someway (doing lots of hashes) does not makes it a good target for Proxmox VE or the like.


Where order it?


A Raspberry just really does not compare with a modern XEON processor, I worked with both quite a bit.
Ok ok, but ARM is becoming popular, VMware has noticied it and they're developing in ARM branch, I don't want that my favorite system (Proxmox) keep stucked only in x86 :p.

You can order here https://happyware.com/uk-en/servers/arm-cpu

I've saw a equivalent server but in x86 (same performance, disks...) and it uses kind 2000W - 3000W power supply
this ones use only 1200W (800W-900W in full load) in long term it saves a lot of energy and in the end pay the extra costs.

Well, sorry but I said a simple one and I guess that there aren't Xeons that has consumption of only 5-10W max.
 
Last edited:

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
14,819
695
133
Thanks for the link, we know these Gigabyte servers. A modern AMD Epyc server looks much better suited for virtualization and network defined storage for me. More powerful, more performance on the I/O level (pci-express 4, nvme) and cheaper (better performance for your money and energy consumption).
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
3,678
754
163
South Tyrol/Italy
shop.maurer-it.com
Ok ok, but ARM is becoming popular, VMware has noticied it and they're developing in ARM branch, I don't want that my favorite system (Proxmox) keep stucked only in x86 :p.
We already evaluated ARM 64 back in 2016, I worked on a Gigabyte board with an Applied Micro CPU, which was slow, but we expected that to a certain degree (not many cores and not high clock). Further, I could test a big server from packet.net with Cavium/Marvell ThunderX2 CPUs (96 cores) which were way slower than my local Intel based workstation with 8x2 Cores, for example when doing a Linux kernel compilation from memory (almost purely CPU bound task). So that wasn't a promising start.
We even had a POC for porting PVE 5.x on it, but performance and HW availability where abysmal, so it was paused and not picked up again.

ARM64 HW throws out many cores but has awful interconnects between them, so they just stall each other half the time.
Especially intel has not that many cores, but good interconnects and much higher "instruction per clock" (IPC) throughput, AMD has actually both nowadays, good interconnects and high core count.

I've saw a equivalent server but in x86 (same performance, disks...) and it uses kind 2000W - 3000W power supply
this ones use only 1200W (800W-900W in full load) in long term it saves a lot of energy and in the end pay the extra costs.

Did you actually confirm that? As lots of those look good on paper but then just do not work out.
As said above, I worked with such a ThunderX2 system and while I did not measure actual power consumption of my Workstation (which beat that system) it has a ~600W PSU, and really do not think that was maxed out, CPU is rated at 95W TDP.

Well, sorry but I said a simple one and I guess that there aren't Xeons that has consumption of only 5-10W max.
There are Xeon-D CPUs with 35W, and they can deliver up to 4-5x (ballparking) the performance, so actually more efficient if you break it down to Instruction/Watt. There are other, non Xeon models, which use 10W at 4 cores like the Intel Atom C3436L - just to name an example.

So basically one can find just as efficient HW in AMD64 space, plus that HW has good connections (SATA, PCIe) that actually work (lots of ARM boards with SATA cannot use to boot from them, and m.2 PCIE slots often just work with WLAN cards not NVMe), the ecosystem for peripherals is there, they work with mainline Linux kernel (that got a bit better with ARM in the last years, but still far from good) and options for really powerful Server HW are available to buy "off the shelf" - all that really does not help the argument for an ARM port.

As already mentioned somewhere else in this forum, if I'd put in personal energy for a port, it would be for an actual open HW platform, not yet another proprietary platform like ARM. RISC-V seems like it could get there, but powerful HW with virtualization support does not yet exist.
 
Last edited:

JVini0166

New Member
Feb 19, 2021
4
0
1
18
There are Xeon-D CPUs with 35W, and they can deliver up to 4-5x (ballparking) the performance, so actually more efficient if you break it down to Instruction/Watt. There are other, non Xeon models, which use 10W at 4 cores like the Intel Atom C3436L - just to name an example.
Compared to which ARM processor?

There is an article: https://www.toptal.com/back-end/arm-servers-armv8-for-datacentres

But also my ARM Raspberry Pi 4 8GB has more power than the Xeon D that the costs are equal ;P
Did you actually confirm that? As lots of those look good on paper but then just do not work out.
As said above, I worked with such a ThunderX2 system and while I did not measure actual power consumption of my Workstation (which beat that system) it has a ~600W PSU, and really do not think that was maxed out, CPU is rated at 95W TDP.
Well, my RPi4 consums less than 15W (All working, RAM, SSD, SDCard, Ethernet, CPU...) and one server that I saw in a company works at 100W PSU only and it's more powerful than a Xeon E5 v3

Although, I guess even the developing for ARM branch be slow I guess it will important in the market, from 2019 to 2021 the market of ARM processors increased very much.

And also there are many motives to create a build to ARM, Microsoft is developing Windows 10 ARM, you can download and test it, many companies are developing their programs to ARM like Adobe, the Apple is already testing a consumer version of ARM of MacOS.

https://siliconangle.com/2020/06/26/exiting-x86-apple-microsoft-embracing-arm-based-pc/

I guess VMware doesn't would waste the time developing their Hypervisor in ARM if it doesn't be worth.

https://virtualizationreview.com/articles/2020/10/13/esxi-arm.aspx
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!