What is the best way to implement high capacity redundant and/or distributed storage in a cluster of miny PCs?

[kayson]

I recently picked up 4 mini PCs that I'm going to use for a proxmox HA cluster:

• i5-8500T
• 32GB RAM
• 10Gb NIC
• 1TB M.2 SSD (glusterfs or ceph for distributed vm storage)
• 128GB M.2 SSD (boot)

I also have ~22TB of bulk data that needs a home. My current plan is to build the following zfs NAS and serve files over kerberized nfs4:

• Ryzen 3 3100
• 64GB ECC RAM
• 4x 16TB SATA HDD (2x mirror vdevs)
• 2x 1TB M.2 SSD (special vdev)
• 10Gb NIC

I'm not super keen on this setup for two reasons: my electricity is very expensive and its an extra machine, and there is no redundancy in the NAS cpu/ram/mobo/psu. With the cluster, I can easily have a node go down or take it down and not have to worry.

This got me thinking, I have 4 SATA drives and 4 nodes, why not add one drive to each node and do some kind of distributed setup that way? I could also potentially swap the 128GB boot drive for another 1TB drive for the bulk data storage.

The problems I see with this are that I have much less RAM on a given node than the NAS, and there's no ECC. I'm also not totally sure how exactly I would/should set up the distributed storage. I was thinking zfs on a VM over iscsi on the nodes' bare metal, but maybe a proxmox ceph volume as a VM storage backing is a better option.

What is a good way of setting this up? I'm very new to this so any help or guidance is greatly appreciated!

 

[LnxBil]

If you want to minimize energy usage, just use one machine for all. You seldomly can optimize for all your goals. If your HA solution is more important, use the 4 machines with CEPH (4 is not a good number for cluster, 3 or 5 would be better).