proxmox on arm64

xiaolin0199

Active Member
Oct 16, 2018
36
1
28
39
Dear all:

In recent years, with the development of arm in the field of server, everyone is very concerned about the progress of proxmox in this field.

I have seen some old posts and I understand your company's ideas. But the actual situation is that some companies or individuals still want to try to work in this area for some reason, and I believe that your company has accumulated considerable experience in this area.

Can you share some information to help everyone?

For example: what components are necessary to build an environment where basic functions are available? Can you provide some useful reference documents? .....

We would appreciate it!
 
Last edited:
  • Like
Reactions: Tmanok

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
973
173
In recent years, with the development of arm in the field of server, everyone is very concerned about the progress of proxmox in this field.

Really? Not in my world (Europe). Its more or less impossible for most to purchase a stable and powerful arm server which is on the same performance level as widely used amd64 hardware. Seem this is different in China.?

As soon as this change, we are prepared to release Proxmox VE for arm64 (most work is already done).
 

xiaolin0199

Active Member
Oct 16, 2018
36
1
28
39
Really? Not in my world (Europe). Its more or less impossible for most to purchase a stable and powerful arm server which is on the same performance level as widely used amd64 hardware. Seem this is different in China.?

Yes, do you know Huawei? Kunpeng920 was released in the second half of last year。And based on this, launched a high-performance server(TaiShan) suitable for different scenarios。

As soon as this change, we are prepared to release Proxmox VE for arm64 (most work is already done).

Is there an approximate time frame? We really look forward to it!
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
973
173
Yes, do you know Huawei? Kunpeng920 was released in the second half of last year。And based on this, launched a high-performance server(TaiShan) suitable for different scenarios。



Is there an approximate time frame? We really look forward to it!

I know Huawei, of course. Send us a bunch of such servers with Kunpen920 to our testlab and we start our testing.

Based on my knowledge our hardware team failed to order such boxes. Just not in the market here for purchase.
 
  • Like
Reactions: gurubert

xiaolin0199

Active Member
Oct 16, 2018
36
1
28
39
I know Huawei, of course. Send us a bunch of such servers with Kunpen920 to our testlab and we start our testing.

Based on my knowledge our hardware team failed to order such boxes. Just not in the market here for purchase.


Oh, we are not Huawei, sorry :(

If your company releases an arm version of pve in the future, we will be happy to try and provide timely feedback on relevant information。
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
973
173
Oh, we are not Huawei, sorry :(

If your company releases an arm version of pve in the future, we will be happy to try and provide timely feedback on relevant information。

We cannot create and test Proxmox VE as a working solution for hardware which just exists on PDF papers ...

I hope you understand the point.
 
  • Like
Reactions: GastonJ

alexskysilk

Renowned Member
Oct 16, 2015
906
126
68
Chatsworth, CA
www.skysilk.com
Yes, do you know Huawei? Kunpeng920 was released in the second half of last year。And based on this, launched a high-performance server(TaiShan) suitable for different scenarios。
I have never seen these actually available to buy; it is my understanding that these indigenous designs exist in China solely for government contracts, and that their price/performance is really poor relative to other available designs. If they are shipping to the public it would be a most interesting development- can you link a retail product that is purchasable?
 

polinet

New Member
Apr 11, 2020
3
3
3
30
It's likely see such arm servers from HP, which has the Proliant m400; but they aren't popular. It's like ARM it's stuck on SoC chips.
In other hands, is it possible to run LXC on Raspberry ? Maybe could be a change to make some sort of Proxmox to Raspberry limited to LXC; in addition to the hability of third vote on quorum
 

Yangzhe

New Member
May 15, 2020
3
0
1
28
Really? Not in my world (Europe). Its more or less impossible for most to purchase a stable and powerful arm server which is on the same performance level as widely used amd64 hardware. Seem this is different in China.?

As soon as this change, we are prepared to release Proxmox VE for arm64 (most work is already done).

What can I do to get Proxmox VE for arm64 with the source code? Can you give me some help? Thanks a lot.
 

mattlach

Active Member
Mar 23, 2016
162
16
38
Boston, MA
We cannot create and test Proxmox VE as a working solution for hardware which just exists on PDF papers ...

I hope you understand the point.

I am not familiar with the Chinese designs, but I do recall reading a lot of reviews of the Cavium (now Marvell owned) ARMv8 ThunderX2 servers and workstations about two years ago.

https://www.servethehome.com/cavium-thunderx2-review-benchmarks-real-arm-server-option/

https://www.anandtech.com/show/12694/assessing-cavium-thunderx2-arm-server-reality


Gigabyte seems to sell them, but I have never tried to order them, as I am more of a home enthusiast. I get the tech once it is decommissioned and cheap on eBay :p


Server:
https://www.gigabyte.com/ARM-Server/R181-T92-rev-100#ov


There is even a Gigabyte Development Workstation:
https://www.anandtech.com/show/12571/gigabyte-thunderxstation-cavium-thunderx2-socs


I don't think ARM is quite ready on the server for me yet, as I have a ton of VM's and containers with software that needs to run, and not all of it is available ported, but I am hoping by the time these machines start showing up used on eBay in my price range, maybe the market has moved in that direction so that I have more options to choose from than the regular x86 suspects.

I'll probably still buy x86 for the next several generations, but having options is good.

I wish RISCV were the next big ISA in computers instead of ARM. Would be nice to have an ISA that is license free for a change...

Would stink to make an arch change just to go from one license hell (Intel, AMD and the x86 and AMD64 licenses) to another license hell (ARM Holdings has been good to date, but who knows under Nvidia ownership. That company has a long history of lock-ins, lock outs and manipulation). I would love to see a truly open ISA in which any chip company that wants to can compete freely without a license.
 
  • Like
Reactions: Tmanok

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
5,569
1,838
174
South Tyrol/Italy
shop.proxmox.com
I am not familiar with the Chinese designs, but I do recall reading a lot of reviews of the Cavium (now Marvell owned) ARMv8 ThunderX2 servers and workstations about two years ago.
I had the opportunity to use such a system, with dual ThunderX2 cores at about that time, a kernel compile on its 96 cores was multiples times slower than a 16 core Intel (non server) CPU, and that is a workload which can be parallelized really really good.

Most high core count ARMs, while sounding good on paper, just cannot schedule good, lack good core interconnects which don't get congested by the simplest things.

We also have a Gigabyte based ARM workstation here, it was a PITA to get it booted, docs and the pre-shipped u-boot were rather lacking, to say the least. It got then flashed with an OVMF based UEFI implementation, which helped, but even then booting a kernel worked with one build, and the stopped working with the next.

It may get better someday, maybe, but that what is available, and I mean really available for the common people and companies, and reasonable priced is rather lacking - you probably still get the best results from a Raspberry Pi and spin-offs, and those are just too limited for a full PVE (and has the same issues with kernel availability, closed ecosystem, ...).

Believe me, we invested quite some resources into this topic, and even the very optimistic devs here got burned too much to keep their optimism.
If I personally start any effort again, then for an actual open architecture.

But that naturally does not have to stop anybody from the community to spearhead the effort of porting to a proprietary arch :)
 

kenji

New Member
Feb 24, 2021
12
4
3
38
I got an ubuntu focal running on my m1 arm macbook air with https://github.com/gyf304/vmcli
Does developing on macOS (ok... proprietary hardware and software, sorry) using a virtual machine would be easier ?
One ubuntu (or debian) VM to compile the kernel, and copying it on the host to boot it in another VM
 
  • Like
Reactions: Tmanok

trayn

New Member
Apr 29, 2021
2
0
1
31
hi guys,

i use proxmox on x86 hardware and i love it, thank you for your great work!
i'm very interested in a arm64 image for the RaspberryPi4 (8GB), too! I know it's not for a business use case with a Pi but maybe a first step to the ARM-architecture and servers.

I followed the ARM-threads since one year but you can't notice me :D

so here is my +1 for the arm support! ;)
 

Tmanok

Active Member
Hello everyone,

With the release of the 10G-BaseT M1 MacMini, a machine that ranks pretty high amongst CPUs: (within the top 300 consumer/server CPUs worldwide)
Intel Core i9-10880H @ 2.30GHz815,3902,77745FCBGA1440Laptop
Intel Xeon Silver 4214Y @ 2.20GHz1215,2481,79085FCLGA3647Server
Intel Xeon Silver 4116T @ 2.10GHz1215,1871,78185FCLGA3647Server
Intel Xeon Silver 4210R @ 2.40GHz1015,1211,884100FCLGA3647Server
Intel Xeon D-2166NT @ 2.00GHz1215,1051,75185FCBGA2518Server
Intel Core i9-9980HK @ 2.40GHz815,0582,67445FCBGA1440Laptop
Intel Xeon E5-2687W v3 @ 3.10GHz1015,0491,929160LGA2011-v3Server
Apple M1 8 Core 3200 MHz815,0373,73415.1UnknownDesktop, Laptop
AMD EPYC 73011614,9911,363155SP3Server
AMD Ryzen 7 PRO 2700814,9172,19965AM4Desktop
Intel Xeon E5-2699A v4 @ 2.40GHz2214,9171,037145FCLGA2011-3Server

I'd like to consider that we will see an M2 chip in a year or two, possibly with more cores, more memory, and more I/O bandwidth. Even as an 8-CPU core, 8-GPU core, and 16-NeuralPU core machine, it is rather desirable. The lack of RAM (max 16GB) is somewhat pitiful, but with that 10Gbps NIC and Thunderbolt ports it could make a low power shared storage server or an edge HV cluster. Perhaps using the M1 as a template/stepping stone, PVE would be prepared to take advantage of other (relatively) cheap <2L computers for edge clustering and I suppose homelabs.

Thanks, simply sharing my thoughts.

Tmanok
 
Mar 16, 2019
289
50
33
61
California, USA
So you're thinking Apple is going to see the light and start supporting Linux on their products? Or they're going to start selling the SoC to third parties? Or what?

Because nothing about M1 makes it easier to support any of the crowd of ARM SoC's that are out there, all different.
 
Last edited:

Tmanok

Active Member
So you're thinking Apple is going to see the light and start supporting Linux on their products? Or they're going to start selling the SoC to third parties? Or what?

Because nothing about M1 makes it easier to support any of the crowd of ARM SoC's that are out there, all different.
Apple? No no definitely not, they will continue to raise their noses at us. But as Linux is progressively moving closer and closer to M1 Support, it will very likely make M2 support quicker and easier. I think that if we are going to take advantage of the hardware being released right now, we need to start development and build knowledge on what is currently available. It is only a matter of time before ARM becomes a significant player in the server space, so why would we not start ahead of the curve and be ready for when that time comes?

Apple is pushing ever so delicately into the server space with the new Mac Pro by the way... Less ridiculous RAM placement, out of band management, and redundant PSU are the few missing pieces there. Anyway I look forward to real server manufacturers making ARM and RISC chips soon, and at the same time, I'm anxious to see the hardware not go to waste because the software is too far behind.
 
Mar 16, 2019
289
50
33
61
California, USA
It is only a matter of time before ARM becomes a significant player in the server space, so why would we not start ahead of the curve and be ready for when that time comes?
"ARM support" in the sense you're discussing already exists for Linux. Nearly every phone has an ARM CPU with a Linux kernel running on it. Debian supports a bunch of ARM boards with their thousands of packages. The instruction set is handled.

The rest of the system is the issue. Bunch of different boot methods (some Raspberry Pi boards boot from the GPU!), locked boot firmware, no standardization of buses or peripherals, lots of proprietary drivers, etc. Why do you think the Android ROM scene is so disorganized? Think about why, say, LineageOS only supports a relative handful of phones and it takes years to add a new one.

Once you understand that you will know why Proxmox is taking a wait and see attitude on ARM.
 
Last edited:
  • Like
Reactions: gurubert and jimi

gwdsl

New Member
Jun 16, 2021
3
1
3
32
I would also love it, if someone could provide instructions on how to compile Proxmox from source on a Apple M1 machine
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!