Ceph/RAID 5 in a small homelab

[sysrq]

Hello,

I m planning to buy a third node for my proxmox homelab cluster. 3 nodes, each one of them will have a secondary ssd for separate storage.
I want some redundancy between those ssds, initially i thought Ceph is my way to go, but after some research i learned that Ceph is designed for enterprise clusters. For my setup with 3 secondary 1 tb ssds, from 3 tb in total i will only have 1 tb usable for data. That’s less that i was expecting.

Any suggestions for redundancy for my data in this small homelab setup? RAID 5 maybe? I m open to new suggestions.

Update: i found out that raid is possible at the node level only, not across nodes..

 

[atuin]

Hi you can try linstor/drbd with replica 1 you can use max 2tb

 

[sysrq]

I did some research and for my needs your suggestion seems to be least expensive and more than enough. Thank you, i will stick with drbd.

 

[LnxBil]

Another approach would be to have two independent PVE nodes and use ZFS replication. You can use redundancy (mirror or RAID5/6 equivalents) locally.

 

[sysrq]

Locally isn't possible, i use some mini pcs as my nodes, i don't have enough expansion slots to add 2 or more ssds to a single node. I have 3 independent nodes in a cluster and 2 of them have 1 tb secondary sata 3 ssd. And i want redundancy in case of a ssd failure on these 2 for my nextcloud data.

So this ZFS replication from my understanding does the same thing as DRBD. Live replication of data from one ssd to another ssd across 2 nodes, right? Exactly the same data on both ssds, if 1 of them fails the other one takes over and keeps the data available.
What is the difference between them, pros and cons?

 

[Geco-iT]

Hi, if only for replicate nextcloud data (one vmdisk), I would go for ZFS sync (proxmox integrated and more simple)

 

[Johannes S]

Hello,

as you already noted Ceph in a small homelab opens a whole can of worms: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

With your current hardware you have basically following options:
- Build a cluster out of your Mini-PCs and use Storage Replication with ZFS: https://pve.proxmox.com/wiki/Storage_Replication
- Leave each node as single-node instance without building a cluster from them ( so each node will still work even if the two other ones are down, clusters are cool but make stuff more complicated and in a Homelab the needed additional networking hardware often enough isn't worth it). Then use pve-zsync to replicate vms between the nodes: https://pve.proxmox.com/wiki/PVE-zsync
- Setup a NAS with the three disks and use NFS or ISCSI to provide storage to your nodes from the NAS

The first two options won't give you much more space (because you will still need to save the data on at least two nodes) , the third one will have worse performance than local storage. And any setup with RAIDZ (ZFS-Jargon for RAID3/RAID5 etc) won't be great for performance either: https://forum.proxmox.com/threads/fabu-can-i-use-zfs-raidz-for-my-vms.159923/

How much storage is on your boot disc? If you have two SSDs of equal size in your nodes I wouldn't bother to seperate OS and data but build a ZFS mirror (aka RAID1 in HW RAID Jargon) from both SSDs and install everything on it. This will give you a better read performance and redundancy for your OS install and VMs.

Another option might be to rethink your whole approach and setup things this way:
- Use one of the Mini-PCs as a ProxmoxBackupServer and (if you build a cluster) qdevice: https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support
- Use the two remaining Mini-PCs as ProxmoxVE Servers for VMS and LXCs. You could build a cluster out of them (then you will need the qdevice!) and have High Availability together with ZFS storage replication and the Backups on the PBS. Or you setup each one as a single-node install (nothing wrong with that) and just backup everything to the PBS. The second variant will give you more space but you will have a longer downtime in case of a restore (PBS allows launching a VM and restoring it on the same time at the cost of performance) a. The first variant will give you a smaller downtime and an additional copy of your VM.


As you see everything depends on what you want to achieve with your setup and which potential downsides you can live with

 

[sysrq]

On my boot disks i have:
Node 1: 128GB m2 sata 3 ssd;
Node 2: 256GB sata 3 SSD;
Node 3: 256GB sata 3 SSD.

I already have the cluster for the 3 nodes and i think i will be fine with zfs replication/drbd, 2 1TB SSDS and only 1 TB usable. I will get one node failure redundancy and i will keep experimenting a little bit with HA (that was the reason behind buying the third node beside the need for extra resources).

Yes, in the future i will add a NAS to my homelab but these are pretty expensive for my budget at the moment.

ZFS replication works in a cluster setup too, right? I am asking that because i see "independent node"term in both of your posts before ZFS replication. 2 ZFS pools for the 2 secondary 1 tb SSDS (different nodes) and ZFS/DRBD replication between them.