proxmox on arm64

[xiaolin0199]

Dear all:

In recent years, with the development of arm in the field of server, everyone is very concerned about the progress of proxmox in this field.

I have seen some old posts and I understand your company's ideas. But the actual situation is that some companies or individuals still want to try to work in this area for some reason, and I believe that your company has accumulated considerable experience in this area.

Can you share some information to help everyone?

For example: what components are necessary to build an environment where basic functions are available? Can you provide some useful reference documents? .....

We would appreciate it！

 

[tom]

In recent years, with the development of arm in the field of server, everyone is very concerned about the progress of proxmox in this field.

Really? Not in my world (Europe). Its more or less impossible for most to purchase a stable and powerful arm server which is on the same performance level as widely used amd64 hardware. Seem this is different in China.?

As soon as this change, we are prepared to release Proxmox VE for arm64 (most work is already done).

 

[xiaolin0199]

Really? Not in my world (Europe). Its more or less impossible for most to purchase a stable and powerful arm server which is on the same performance level as widely used amd64 hardware. Seem this is different in China.?

Yes, do you know Huawei? Kunpeng920 was released in the second half of last year。And based on this, launched a high-performance server（TaiShan） suitable for different scenarios。

As soon as this change, we are prepared to release Proxmox VE for arm64 (most work is already done).

Is there an approximate time frame? We really look forward to it!

 

[tom]

Yes, do you know Huawei? Kunpeng920 was released in the second half of last year。And based on this, launched a high-performance server（TaiShan） suitable for different scenarios。



Is there an approximate time frame? We really look forward to it!

I know Huawei, of course. Send us a bunch of such servers with Kunpen920 to our testlab and we start our testing.

Based on my knowledge our hardware team failed to order such boxes. Just not in the market here for purchase.

 

[xiaolin0199]

I know Huawei, of course. Send us a bunch of such servers with Kunpen920 to our testlab and we start our testing.

Based on my knowledge our hardware team failed to order such boxes. Just not in the market here for purchase.

Oh, we are not Huawei, sorry

If your company releases an arm version of pve in the future, we will be happy to try and provide timely feedback on relevant information。

 

[tom]

Oh, we are not Huawei, sorry

If your company releases an arm version of pve in the future, we will be happy to try and provide timely feedback on relevant information。

We cannot create and test Proxmox VE as a working solution for hardware which just exists on PDF papers ...

I hope you understand the point.

 

[alexskysilk]

Yes, do you know Huawei? Kunpeng920 was released in the second half of last year。And based on this, launched a high-performance server（TaiShan） suitable for different scenarios。

I have never seen these actually available to buy; it is my understanding that these indigenous designs exist in China solely for government contracts, and that their price/performance is really poor relative to other available designs. If they are shipping to the public it would be a most interesting development- can you link a retail product that is purchasable?

 

[polinet]

It's likely see such arm servers from HP, which has the Proliant m400; but they aren't popular. It's like ARM it's stuck on SoC chips.
In other hands, is it possible to run LXC on Raspberry ? Maybe could be a change to make some sort of Proxmox to Raspberry limited to LXC; in addition to the hability of third vote on quorum

 

[Yangzhe]

Really? Not in my world (Europe). Its more or less impossible for most to purchase a stable and powerful arm server which is on the same performance level as widely used amd64 hardware. Seem this is different in China.?

As soon as this change, we are prepared to release Proxmox VE for arm64 (most work is already done).

What can I do to get Proxmox VE for arm64 with the source code? Can you give me some help? Thanks a lot.

 

[t.lamprecht]

https://git.proxmox.com/
https://git.proxmox.com/?p=pve-comm...bcf17474b2fe67b8258b9f30c221d388c38e2;hb=HEAD

 

[mattlach]

We cannot create and test Proxmox VE as a working solution for hardware which just exists on PDF papers ...

I hope you understand the point.

I am not familiar with the Chinese designs, but I do recall reading a lot of reviews of the Cavium (now Marvell owned) ARMv8 ThunderX2 servers and workstations about two years ago.

https://www.servethehome.com/cavium-thunderx2-review-benchmarks-real-arm-server-option/

https://www.anandtech.com/show/12694/assessing-cavium-thunderx2-arm-server-reality


Gigabyte seems to sell them, but I have never tried to order them, as I am more of a home enthusiast. I get the tech once it is decommissioned and cheap on eBay


Server:
https://www.gigabyte.com/ARM-Server/R181-T92-rev-100#ov


There is even a Gigabyte Development Workstation:
https://www.anandtech.com/show/12571/gigabyte-thunderxstation-cavium-thunderx2-socs


I don't think ARM is quite ready on the server for me yet, as I have a ton of VM's and containers with software that needs to run, and not all of it is available ported, but I am hoping by the time these machines start showing up used on eBay in my price range, maybe the market has moved in that direction so that I have more options to choose from than the regular x86 suspects.

I'll probably still buy x86 for the next several generations, but having options is good.

I wish RISCV were the next big ISA in computers instead of ARM. Would be nice to have an ISA that is license free for a change...

Would stink to make an arch change just to go from one license hell (Intel, AMD and the x86 and AMD64 licenses) to another license hell (ARM Holdings has been good to date, but who knows under Nvidia ownership. That company has a long history of lock-ins, lock outs and manipulation). I would love to see a truly open ISA in which any chip company that wants to can compete freely without a license.

 

[t.lamprecht]

I am not familiar with the Chinese designs, but I do recall reading a lot of reviews of the Cavium (now Marvell owned) ARMv8 ThunderX2 servers and workstations about two years ago.

I had the opportunity to use such a system, with dual ThunderX2 cores at about that time, a kernel compile on its 96 cores was multiples times slower than a 16 core Intel (non server) CPU, and that is a workload which can be parallelized really really good.

Most high core count ARMs, while sounding good on paper, just cannot schedule good, lack good core interconnects which don't get congested by the simplest things.

Server:
https://www.gigabyte.com/ARM-Server/R181-T92-r

Server:
https://www.gigabyte.com/ARM-Server/R181-T92-rev-100#ov


There is even a Gigabyte Development Workstation:
https://www.anandtech.com/show/12571/gigabyte-thunderxstation-cavium-thunderx2-socs
ev-100#ov


There is even a Gigabyte Development Workstation:
https://www.anandtech.com/show/12571/gigabyte-thunderxstation-cavium-thunderx2-socs

We also have a Gigabyte based ARM workstation here, it was a PITA to get it booted, docs and the pre-shipped u-boot were rather lacking, to say the least. It got then flashed with an OVMF based UEFI implementation, which helped, but even then booting a kernel worked with one build, and the stopped working with the next.

It may get better someday, maybe, but that what is available, and I mean really available for the common people and companies, and reasonable priced is rather lacking - you probably still get the best results from a Raspberry Pi and spin-offs, and those are just too limited for a full PVE (and has the same issues with kernel availability, closed ecosystem, ...).

Believe me, we invested quite some resources into this topic, and even the very optimistic devs here got burned too much to keep their optimism.
If I personally start any effort again, then for an actual open architecture.

But that naturally does not have to stop anybody from the community to spearhead the effort of porting to a proprietary arch

 

[kenji]

I got an ubuntu focal running on my m1 arm macbook air with https://github.com/gyf304/vmcli
Does developing on macOS (ok... proprietary hardware and software, sorry) using a virtual machine would be easier ?
One ubuntu (or debian) VM to compile the kernel, and copying it on the host to boot it in another VM

 

[trayn]

hi guys,

i use proxmox on x86 hardware and i love it, thank you for your great work!
i'm very interested in a arm64 image for the RaspberryPi4 (8GB), too! I know it's not for a business use case with a Pi but maybe a first step to the ARM-architecture and servers.

I followed the ARM-threads since one year but you can't notice me

so here is my +1 for the arm support!

 

[kenji]

a pi 4 cluster for playing could be very very cool

 

[Tmanok]

Hello everyone,

With the release of the 10G-BaseT M1 MacMini, a machine that ranks pretty high amongst CPUs: (within the top 300 consumer/server CPUs worldwide)

	Intel Core i9-10880H @ 2.30GHz	8	15,390	2,777	45	FCBGA1440	Laptop
	Intel Xeon Silver 4214Y @ 2.20GHz	12	15,248	1,790	85	FCLGA3647	Server
	Intel Xeon Silver 4116T @ 2.10GHz	12	15,187	1,781	85	FCLGA3647	Server
	Intel Xeon Silver 4210R @ 2.40GHz	10	15,121	1,884	100	FCLGA3647	Server
	Intel Xeon D-2166NT @ 2.00GHz	12	15,105	1,751	85	FCBGA2518	Server
	Intel Core i9-9980HK @ 2.40GHz	8	15,058	2,674	45	FCBGA1440	Laptop
	Intel Xeon E5-2687W v3 @ 3.10GHz	10	15,049	1,929	160	LGA2011-v3	Server
	Apple M1 8 Core 3200 MHz	8	15,037	3,734	15.1	Unknown	Desktop, Laptop
	AMD EPYC 7301	16	14,991	1,363	155	SP3	Server
	AMD Ryzen 7 PRO 2700	8	14,917	2,199	65	AM4	Desktop
	Intel Xeon E5-2699A v4 @ 2.40GHz	22	14,917	1,037	145	FCLGA2011-3	Server

I'd like to consider that we will see an M2 chip in a year or two, possibly with more cores, more memory, and more I/O bandwidth. Even as an 8-CPU core, 8-GPU core, and 16-NeuralPU core machine, it is rather desirable. The lack of RAM (max 16GB) is somewhat pitiful, but with that 10Gbps NIC and Thunderbolt ports it could make a low power shared storage server or an edge HV cluster. Perhaps using the M1 as a template/stepping stone, PVE would be prepared to take advantage of other (relatively) cheap <2L computers for edge clustering and I suppose homelabs.

Thanks, simply sharing my thoughts.

Tmanok

 

[BobhWasatch]

So you're thinking Apple is going to see the light and start supporting Linux on their products? Or they're going to start selling the SoC to third parties? Or what?

Because nothing about M1 makes it easier to support any of the crowd of ARM SoC's that are out there, all different.

 

[Tmanok]

So you're thinking Apple is going to see the light and start supporting Linux on their products? Or they're going to start selling the SoC to third parties? Or what?

Because nothing about M1 makes it easier to support any of the crowd of ARM SoC's that are out there, all different.

Apple? No no definitely not, they will continue to raise their noses at us. But as Linux is progressively moving closer and closer to M1 Support, it will very likely make M2 support quicker and easier. I think that if we are going to take advantage of the hardware being released right now, we need to start development and build knowledge on what is currently available. It is only a matter of time before ARM becomes a significant player in the server space, so why would we not start ahead of the curve and be ready for when that time comes?

Apple is pushing ever so delicately into the server space with the new Mac Pro by the way... Less ridiculous RAM placement, out of band management, and redundant PSU are the few missing pieces there. Anyway I look forward to real server manufacturers making ARM and RISC chips soon, and at the same time, I'm anxious to see the hardware not go to waste because the software is too far behind.

 

[BobhWasatch]

It is only a matter of time before ARM becomes a significant player in the server space, so why would we not start ahead of the curve and be ready for when that time comes?

"ARM support" in the sense you're discussing already exists for Linux. Nearly every phone has an ARM CPU with a Linux kernel running on it. Debian supports a bunch of ARM boards with their thousands of packages. The instruction set is handled.

The rest of the system is the issue. Bunch of different boot methods (some Raspberry Pi boards boot from the GPU!), locked boot firmware, no standardization of buses or peripherals, lots of proprietary drivers, etc. Why do you think the Android ROM scene is so disorganized? Think about why, say, LineageOS only supports a relative handful of phones and it takes years to add a new one.

Once you understand that you will know why Proxmox is taking a wait and see attitude on ARM.

 

[gwdsl]

I would also love it, if someone could provide instructions on how to compile Proxmox from source on a Apple M1 machine