proxmox on arm64
- Thread starter xiaolin0199
- Start date
-
- Tags
- arm64
The proxmox team is earning money by selling licences to big companies that are running alot of big servers. If I were a big company that pays alot of money I would want that the money is invested in developing software for hardware that can actually being used in a data center. So Raspis or M1s aren't really a thing worth to support. We need big racks of enterprise grade ARM servers first.
I agree with you for a production use, but as a developer, being able to have a proxmox (or even a cluster) right inside my (m1) macbook is really interesting, and having the same on my desk with few raspberry pis can allow to do some cluster testing easily (switch off one Pi for example)
nice catch, so now I need to be able to run a Debian Official Cloud Images using Apple Virtualization framework (both intel and arm), as used here :
https://github.com/evansm7/vftool
and here
https://github.com/gyf304/vmcli
And finally follow the guide linked by @fpausp :
Code:
root@armVM:~$ curl https://gitlab.com/minkebox/pimox/-/raw/master/dev/KEY.gpg | apt-key add -
root@armVM:~$ curl https://gitlab.com/minkebox/pimox/-/raw/master/dev/pimox.list > /etc/apt/sources.list.d/pimox.list
root@armVM:~$ apt update
root@armVM:~$ apt install pve-managerHello,
I am also interested in using proxmox on ARM servers.
I run my projects on pine64 devices and they have release a cheap 8GB single board computer with QuadCore Cortex-A55 2Ghz CPU in dev stage.
I have already ordered a couple and I am targeting this device to run arm64 cluster.
This is not a commercial post, I just want to highlight the fact there are great opportunities to use virtualization on new low cost and low power hardware for eco-responsible citizens.
Having a proxmox release able to run a debian or armbian VM would be really helpfull for my projects.
https://pine64.com/product/quartz64-model-a-8gb-single-board-computer/
I am also interested in using proxmox on ARM servers.
I run my projects on pine64 devices and they have release a cheap 8GB single board computer with QuadCore Cortex-A55 2Ghz CPU in dev stage.
I have already ordered a couple and I am targeting this device to run arm64 cluster.
This is not a commercial post, I just want to highlight the fact there are great opportunities to use virtualization on new low cost and low power hardware for eco-responsible citizens.
Having a proxmox release able to run a debian or armbian VM would be really helpfull for my projects.
https://pine64.com/product/quartz64-model-a-8gb-single-board-computer/
What about the Solid Run HoneyComb LX2?
- 16 x A72 cores @ 2.0GHz
- up to 64MB DDR4
- on board 64GB eMMC
- M2 slot
- 1 x PCIe x8 Gen3.0
- 1 x GigE
- 4 x SFP+ 10GigE
- MiniITX form factor
Starting at 750 USD is not bad for the specs.
https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/
- 16 x A72 cores @ 2.0GHz
- up to 64MB DDR4
- on board 64GB eMMC
- M2 slot
- 1 x PCIe x8 Gen3.0
- 1 x GigE
- 4 x SFP+ 10GigE
- MiniITX form factor
Starting at 750 USD is not bad for the specs.
https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/
You need at least cores with the second digit being >= 5, e.g. A75 or A55, as only then, it's modern enough to support the ARM v8.2 spec including stuff like the NEON instruction set, which is a must-have for anything server related, especially clustering in Proxmox VE or hashing and (on-the-wire and at-rest) encryption in Proxmox Backup Server.
FYI, I was interested in efficiency claims a bit at the start of the year and tried to actually come up with a test for that using real workloads, one I do daily (compilation) and one the PBS uses (which is not really a synthetic benchmark, as it uses the exact same methods an actual backup would do), the results is available at:
https://pve.proxmox.com/wiki/PBS_Client_CPU_Benchmark
Key take away:
* a (normal clocked) dual-core from 2008 beat's a Cortex-A72 Quad-core rather easily performance, but not efficiency wise.
* A Xeon E5-2620 v3 from 2014 has 5 times the higher combined throughput while being almost twice as efficient when normalizing for cores and TDP.
* A run-of-the-mill cheap Celeron J4105 (IIRC, ~100$ through an ASRock J4105-ITX) was faster and more efficient in those tests.
I checked around for more modern boards, so I could get an A55+ or A75+ to check out. But the single thing available would have been the NVIDIA tensor flow/pattern recognition ("KI") evaluation boards and I wanted to avoid spending quite some bucks for a lot of GPU/floating point power which I would not use.
Even if it looks somewhat OK spec wise for an entry platform for home lab stuff, I would be surprised if it actually performs better than a 3 to five-year-old amd64 based server, I mean would be expected, the CPU design is from 2016, so def. not bleeding edge anymore.
The 4x 10 GBps look nice, ARM was always quite good at providing fast network interfaces, but the IO interfaces available do not seem like one could actually use all that. Could be OK for a pure networking appliance, e.g., as router or the like though.
FWIW, you can also get something like an ASRock Rack EPYC3101D4I-2T (500 to 600 bucks here) or something similar to a Supermicro X11SDV-4C-TLN2F, yes those have fewer cores than 20, but they will probably still get you about the same amount of actual workload done, and that with way less hassle. And those are only models with embedded passive CPUs, there's much more available if one does not have to restrict themselves to that criteria.
But as mentioned a few times, we already invested a bit of work to make our software not locked to a single platform, so bootstrapping PVE and PBS is already possible (there are even some tutorial floating around), if you think the ARM64 is just worth it for you, you can use it. But it's still not worth the time required for us to provide professional enterprise support for those systems, especially as the best HW available, still gets outperformed by years old entry level amd64 based one.
Ok, this sounds like a real constraint I was not aware of.
Of course x86 is still here and there is nothing against it but I see that x86 is loosing momentum as architecture. No one is asking to forget the x86 architecture right now but rather to have the possibility to get some hands on experience on ARM64 platforms.
Of course an A72 based CPU is not cutting edge (beside the arch constraints above) but not everybody will be able to get an Ampere Altra system...
An official ARM64 community edition (without professional support) with some requirements would be absolutely enough to test and build up experience.
I do not think it makes much sense to release something as official and then not support it, that's IMO contradicting and would mean that users are in limbo on serious issues like data corruption, some HW specific bug or what not, if user run into problems specific to the arm kernel/QEMU/ceph/... we would still need to look into it and thus invest lots of resources, everything else would just shine bad light onto the projects. Also build HW and setups would need to be maintained, and build time is just so slow on those things, we already need an hour for ceph on a high performant x86, I can only imagine that requiring 8+ hours on the Ampere xgene2 we have, would be (somewhat ironically) probably faster to cross compile from our x86 based HW.
The people that really want to do that themselves can do so, and they can already use the official source code for that, that is IMO enough unofficial support.
As said somewhere else, either this thread or some other here, I do not see many, if at all, benefits from ARM64 over amd64/x86_64, both are proprietary, but the latter is at least available in much more quantity, more modern designs and in a broader spectrum regarding different use cases and also established in the server space since decades. Will arm see some gain in traction in the future? Quite possible, but for anybody not rolling out their custom chip and/or getting access to such closed of HW, the incentive to switch is just way too small currently to make it worthwhile.
I also really do not see a dwindling momentum in the amd64/x86_64, ecosystem combining both Intel and AMD's recent quarterly reports, one still sees quite some growth, and they are already huge.
Personally I already stated that only an actual free and open source arch like RISC-V could be worth it, and while there are big steps made in HW available the virtualization extension is not yet fully ratified as standard. A CPU, that actually would provide more freedom, more trust and less hassle as the proprietary management engines (which ARM has too), the µcode mess and whatever else is hidden in arm64 or x86 designs, could be worth the work.
But, as repeated over and over already: trading one proprietary CPU arch for another, just as proprietary but less powerful and with a much smaller server-space ecosystem, seems just not worth it.
Ah ok I see what you're saying. Yes I'm aware of most consumer processors
but I was not familiar with some of the intricacies used by arm hardware for boot processes and drivers. Regarding phone adoption, that's also a big part of the reason why the Ubuntu and Windows phones never really went anywhere... ARM Servers (very broad term I understand) in the enterprise setting for say, cloud infrastructure, are really catching on and probably standardizing massively right now. Will be interesting to see how software stacks like PVE, CloudStack and others grow with the change in cloud CPU architecture.
Tmanok
Rofl...thanks. Didn't know that.
I flashed my old smartphones with cyanogenmod and so on to be able to receive security updates. For example I bought a 800€ flagship smartphone and samsung stopped releasing updates after 8 moths or so...and no device should go online without security fixes and I don't wanted to throw it in the trash after only 8 months. I really hope that will get better with the closed source drivers, so everyone can just use some opensource andoids that will get security updates over many years, because smartphone manufacturers won't start increasing the support lenght if they aren't forced to do that by the goverments...
And the Android custom roms never worked really well. There were always some problems like cams that lost focus and so on because all drivers are closed source so they needed to reverse engineer them...
I really hope that will work better with server. Atleast it seems that enterprise grade stuff got better linux support and more drivers are actually open source or atleast better documented.
Last edited:
Finally got a debian lxc container running on my m1 macbook : https://forum.proxmox.com/threads/how-to-run-pve-7-on-a-raspberry-pi.95658/post-425043
Last edited:
Awesome! Good job man.
In that same line of thought, the new M1 Pro and M1 Max came out and according to Apple they will be somewhere in the 30,000 on the benchmark score which is 1.5x my Dell XPS 15's i9-11900H. I gotta hand it to Apple, if this keeps up and they release a gen3 M1 MacMini I might be forced to make a Mini-Cluster as my homelab rofl.
Imagine a MacMini with: 64GB DDR5 per node, an absurd amount of GPU and Neural Cores, PCIe Gen4 SSD, and likely 4x Thunderbolt 4 ports. I could build a Ceph cluster with Thunderbolt drive arrays and Thunderbolt to 40Gbps networking and the whole thing could probably fit in a filing cabinet drawer. It's difficult to state how excited I am about these kinds of prospects, I'm starting to really come around to the idea of an ARM PVE within the next 3-5 years.
Oh and don't forget the 200GB/s and 400GB/s memory bandwidth. Heckin' doodle, that's nuts.
In that same line of thought, the new M1 Pro and M1 Max came out and according to Apple they will be somewhere in the 30,000 on the benchmark score which is 1.5x my Dell XPS 15's i9-11900H. I gotta hand it to Apple, if this keeps up and they release a gen3 M1 MacMini I might be forced to make a Mini-Cluster as my homelab rofl.
Imagine a MacMini with: 64GB DDR5 per node, an absurd amount of GPU and Neural Cores, PCIe Gen4 SSD, and likely 4x Thunderbolt 4 ports. I could build a Ceph cluster with Thunderbolt drive arrays and Thunderbolt to 40Gbps networking and the whole thing could probably fit in a filing cabinet drawer. It's difficult to state how excited I am about these kinds of prospects, I'm starting to really come around to the idea of an ARM PVE within the next 3-5 years.
Oh and don't forget the 200GB/s and 400GB/s memory bandwidth. Heckin' doodle, that's nuts.
Last edited:
Thanks, but all credits should go to @Tux_1024 (I only `apt install proxmox-ve` what he packaged), then got issues with apparmor (I'm have to make my homework on this)
Excited too, and 64GB macbooks are near already here ^^ (okay, letting a 120hz display closed isn't a good idea), and the "Mac Pro" and iMac Pro hasn't made the transition to m1 (or m2...)
Maybe on a single 64Gb macbook, you can run a 3 (or more) proxmox inside UTM/qemu to make a "dev cluster"
UTM/qemu can be replaced with apple Virtualization.framework, I'd love to compare performances
That's interesting.
I didn't think that would work, considering LXC shares the kernel with the host, and MacOS does not use a Linux kernel.
The linux kernel is running within a virtual machine with UTM (https://mac.getutm.app)