Proxmox for Raspberry Pi 4 with Ceph

[ben90818532]

Imagine:

A cluster of Raspberry Pi 4's each with 4gb of RAM and full Gigabit capable of running LXC containers with HA failover and Ceph storage.

Use of ZRAM could allow around 2.5 to 3gb of actual container memory usage per Raspberry Pi 4 node, limiting Ceph bandwidth to say 800mbps could allow reasonable sharing of the 1gbps nic beween both containers and Ceph.

I think this could be a great way to get more students, hobbyists and even professionals interested in and learning about Proxmox as a whole.

 

[Alwin]

You can tinker around but it will not be really usable. Most of the services and feature would consume all memory, let alone running any LXC or ceph.

It is already easy to play around with Proxmox VE + Ceph, just create three VMs on the hypervisor of your choice. Setup and installation work similar to their physical counter parts.

 

[ben90818532]

VMware has already shown a demo of ESXI running on Raspberry Pi

https://youtu.be/UwOhiq5YOJ0

https://youtu.be/d6bN4ayBFWk

 

[Alwin]

The first video is promotional and doesn't show any running ESXi. The second video has a demo with an ESXi and an stripped down, idle VM. But there is no workload running. Also they are speaking about industrial ARM machines, these are far of any Raspberry Pi.

In any case, the hardware of an Raspberry Pi is not intended for VM/CT hosting or SDS. Nevertheless, you can build all Proxmox VE packages for the ARM architecture.

 

[Tmanok]

Good news! 8GB looks very promising. I'm thinking it would do a great as a High Availability container cluster, perhaps for web servers, IOT containers, etc. Anyone remember this great chasis full of older PIs? https://www.raspberrypi.org/forums/viewtopic.php?t=155030

 

[LnxBil]

Good news! 8GB looks very promising. I'm thinking it would do a great as a High Availability container cluster, perhaps for web servers, IOT containers, etc.

I do not expect the I/O to be better. Without proper I/O, you don't need the rest.CEPH without I/O performance is not enjoyable.

 

[Tmanok]

I do not expect the I/O to be better. Without proper I/O, you don't need the rest.CEPH without I/O performance is not enjoyable.

Well perhaps not with CEPH, but since the RPI 3 or 4 now it has included Gigabit Ethernet which is alarmingly better than "Fast Ethernet" (100Mbps). In addition, the USB 3.0 and USB Boot firmware have perhaps made it better off to fulfil the role of a generic proxmox node for containerization or lightweight VMs. So between the NIC serving 1000Mbps and the USB 3.0 ports easily serving 1600Mbps ~ 5000Mbps, I see the I/O performing mighty fine for a cluster of 5 -10 of these 10-20W cards, most especially compared to my 400W~1000W Servers. I might even be able to consume the same wattage and achieve a greater overall performance and redundancy for my gear at a comparable price range....

I'm thinking of doing something similar to this with two empty 1U 1/4 Depth Chassis I have lying about: https://www.raspberrypi.org/forums/viewtopic.php?t=81949

 

[LnxBil]

10-20W cards

Wow, that's huge. My Mini-ITX with a 4-core J4205 and 16 GB-RAM including 2x SSHD 2 TB only consumes 11W when idle and that is much much more computing and foremost I/O power. If you use SSDs, you can even reduce the energy footprint. Don't get me wrong, I love PIs and I have a lot of them, but the I/O is so incredible slow compared to "real" computers, even with USB3 that is shared with all high speed devices.
If you buy a new Pi4-cluster, please benchmark it and share the results.

 

[Tmanok]

Hey LnxBil, they claim 2-3.4 watts idle, the PSU is rated at 15W. I'm assuming your Mini-ITX is probably rated for more than 50-100W? Though I'm interested to hear and see a model number now that you'd peaked my curiousity

It's true that your Mini-ITX is a "real" computer, but my fascination is stuffing 6-20 PIs into a 1U chassis for reliable hot-swappable / ubiquitous hypervisors for my thin LXCs and VMs as I've said earlier. Do send your machine's specs, is it rack mountable?

 

[LnxBil]

Do send your machine's specs, is it rack mountable?

If you put it in a rack mounted case, sure. There exist a least a couple of Dual-Mini-ITX cases. I 3D printed a deskbench and use a Pico PSU, so that I really have a very small footprint.
Running 4 openssl benchmarks (= Cores) jumps to almost 20W. The Pico-PSU is rated up to 90W and a little bit oversized. The Board is a ASRock J4204-ITX (CPU TDP of 10W) with one PCIe x1 and 4 SATA ports, passively cooled, USB3, HDMI, DVI and VGA. IT's pretty great for PVE, as it is the cheapest solution I found for the low power consumption - and I only needed the board, I already had RAM flying around. I also thought of the APU2, but they are very limited with respect to RAM, but rack-mountable with official casing. We ran a highly available firewall on two of those boxes in the days of PVE with a cluster of two nodes and DRBD.

most especially compared to my 400W~1000W Servers.

What are those machines? Top-loader disk monsters?
I just compared that to our PVE cluster and it has roughly 1000W for 5x PVE cluster nodes (HP DL360G6, dual CPU 3.06 GHz) and our Fujitsu DX100S3 SAN with 24 SAS disks. With newer machines and less rotating disks, you can even lower the power consumption.

 

[Tmanok]

I suppose your J4205 (https://www.cpubenchmark.net/cpu.php?cpu=Intel+Pentium+J4205+@+1.50GHz&id=2877) is comparable to an RPI CPU then and that would make sense given your wattage, very impressive given it's nearly the same price as a PI and has 4 SATA ports and PCIe. So aside from the scale I'd say your board is clearly a winner.

My primary nodes are indeed a few HP Proliant G5, G6, G7 machines and IBM System X machine (M3, M4). with only a few hundred GB of DDR3 ECC shared between all my servers. But even then I find it difficult to soak the RAM for most of my projects (Confluence / Emby / Web Scraping / Workstation VMS, being my highest consumers of RAM). We're straying off topic, PM me for more, cheers.

 

[ben90818532]

Looks like the 8gb Pi 4 is now available to order:

https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-now-at-75/

With 8gb of ram there will no doubt be increased interest in running containers and even hypervisors on the Pi 4.

 

[Jannoke]

The first video is promotional and doesn't show any running ESXi. The second video has a demo with an ESXi and an stripped down, idle VM. But there is no workload running. Also they are speaking about industrial ARM machines, these are far of any Raspberry Pi.

In any case, the hardware of an Raspberry Pi is not intended for VM/CT hosting or SDS. Nevertheless, you can build all Proxmox VE packages for the ARM architecture.

I'm new to debian package building, but is there "srpm" style packages available for debian, so that packages can simply be rebuilt without re-making all definitions needed for the debian package system?

 

[LnxBil]

I'm new to debian package building, but is there "srpm" style packages available for debian, so that packages can simply be rebuilt without re-making all definitions needed for the debian package system?

It's the debian subfolder that includes everything.