Ceph/RAID 5 in a small homelab

[sysrq]

Hello,

I m planning to buy a third node for my proxmox homelab cluster. 3 nodes, each one of them will have a secondary ssd for separate storage.
I want some redundancy between those ssds, initially i thought Ceph is my way to go, but after some research i learned that Ceph is designed for enterprise clusters. For my setup with 3 secondary 1 tb ssds, from 3 tb in total i will only have 1 tb usable for data. That’s less that i was expecting.

Any suggestions for redundancy for my data in this small homelab setup? RAID 5 maybe? I m open to new suggestions.

Update: i found out that raid is possible at the node level only, not across nodes..

 

[atuin]

Hi you can try linstor/drbd with replica 1 you can use max 2tb

 

[sysrq]

I did some research and for my needs your suggestion seems to be least expensive and more than enough. Thank you, i will stick with drbd.

 

[LnxBil]

Another approach would be to have two independent PVE nodes and use ZFS replication. You can use redundancy (mirror or RAID5/6 equivalents) locally.

 

[sysrq]

Locally isn't possible, i use some mini pcs as my nodes, i don't have enough expansion slots to add 2 or more ssds to a single node. I have 3 independent nodes in a cluster and 2 of them have 1 tb secondary sata 3 ssd. And i want redundancy in case of a ssd failure on these 2 for my nextcloud data.

So this ZFS replication from my understanding does the same thing as DRBD. Live replication of data from one ssd to another ssd across 2 nodes, right? Exactly the same data on both ssds, if 1 of them fails the other one takes over and keeps the data available.
What is the difference between them, pros and cons?

 

[Geco-iT]

Hi, if only for replicate nextcloud data (one vmdisk), I would go for ZFS sync (proxmox integrated and more simple)

 

[Johannes S]

Hello,

as you already noted Ceph in a small homelab opens a whole can of worms: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

With your current hardware you have basically following options:
- Build a cluster out of your Mini-PCs and use Storage Replication with ZFS: https://pve.proxmox.com/wiki/Storage_Replication
- Leave each node as single-node instance without building a cluster from them ( so each node will still work even if the two other ones are down, clusters are cool but make stuff more complicated and in a Homelab the needed additional networking hardware often enough isn't worth it). Then use pve-zsync to replicate vms between the nodes: https://pve.proxmox.com/wiki/PVE-zsync
- Setup a NAS with the three disks and use NFS or ISCSI to provide storage to your nodes from the NAS

The first two options won't give you much more space (because you will still need to save the data on at least two nodes) , the third one will have worse performance than local storage. And any setup with RAIDZ (ZFS-Jargon for RAID3/RAID5 etc) won't be great for performance either: https://forum.proxmox.com/threads/fabu-can-i-use-zfs-raidz-for-my-vms.159923/

How much storage is on your boot disc? If you have two SSDs of equal size in your nodes I wouldn't bother to seperate OS and data but build a ZFS mirror (aka RAID1 in HW RAID Jargon) from both SSDs and install everything on it. This will give you a better read performance and redundancy for your OS install and VMs.

Another option might be to rethink your whole approach and setup things this way:
- Use one of the Mini-PCs as a ProxmoxBackupServer and (if you build a cluster) qdevice: https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support
- Use the two remaining Mini-PCs as ProxmoxVE Servers for VMS and LXCs. You could build a cluster out of them (then you will need the qdevice!) and have High Availability together with ZFS storage replication and the Backups on the PBS. Or you setup each one as a single-node install (nothing wrong with that) and just backup everything to the PBS. The second variant will give you more space but you will have a longer downtime in case of a restore (PBS allows launching a VM and restoring it on the same time at the cost of performance) a. The first variant will give you a smaller downtime and an additional copy of your VM.


As you see everything depends on what you want to achieve with your setup and which potential downsides you can live with

 

[sysrq]

On my boot disks i have:
Node 1: 128GB m2 sata 3 ssd;
Node 2: 256GB sata 3 SSD;
Node 3: 256GB sata 3 SSD.

I already have the cluster for the 3 nodes and i think i will be fine with zfs replication/drbd, 2 1TB SSDS and only 1 TB usable. I will get one node failure redundancy and i will keep experimenting a little bit with HA (that was the reason behind buying the third node beside the need for extra resources).

Yes, in the future i will add a NAS to my homelab but these are pretty expensive for my budget at the moment.

ZFS replication works in a cluster setup too, right? I am asking that because i see "independent node"term in both of your posts before ZFS replication. 2 ZFS pools for the 2 secondary 1 tb SSDS (different nodes) and ZFS/DRBD replication between them.

 

[LnxBil]

You also have to consider if you can tolerate potential data loss with asynchronous replication with ZFS or if you absolutely need synchronous replication and zero data loss with DRBD. CEPH would also be a zero data loss setup.

For the third node, a simple Raspberry PI would have sufficed if you're not running anything on it. You will be however prepared to have them run CEPH.

 

[sysrq]

I already have the third node, an sh elitedesk without storage expansion slots.
So i can add secondary ssds on the first 2 nodes only (in the future i will look for a NAS for storage), until then i want the best redundancy for the hardware i have. 3 nodes, only one with a secondary 1 tb ssd, i ll buy another 1 tb ssd for the second one, the third one no secondary ssd.

I was thinking DRBD or ZFS rep for the first 2, this way they are protected and redundant. For the third back-ups on node 1/2 and manual restoration in case of a failure.

Yes, synchronous replication, i want all my data to be redundant.
On the future either a NAS either replacing all my main drives with larger ones and configuring ceph with replication of 3 (for the main drives and DRBD/ZFS replication for the first 2 as i said previously). I don’t know the exact plan yet, all i know is that i want redundancy in case of a failure.

 

[LnxBil]

I already have the third node, an sh elitedesk without storage expansion slots.

You don't need PVE to be on an extra disk. You could also split it so that you can use one disk for everything.
It's not optimal, but your hardware setup is currently also not optimal.

On the future either a NAS either replacing all my main drives with larger ones and configuring ceph with replication of 3 (for the main drives and DRBD/ZFS replication for the first 2 as i said previously). I don’t know the exact plan yet, all i know is that i want redundancy in case of a failure.

This would be less redundant than having the data in the cluster. Now at most 1 can fail at any time and you would be still operating. If you NAS would fail, you would not be operating anymore.

 

[sysrq]

You don't need PVE to be on an extra disk. You could also split it so that you can use one disk for everything.
It's not optimal, but your hardware setup is currently also not optimal.

Yes, the third node should have been with an extra expansion slot for storage, ik.
But it's fine, it is a small homelab, not some production environment, i want to learn and experiment.

This would be less redundant than having the data in the cluster. Now at most 1 can fail at any time and you would be still operating. If you NAS would fail, you would not be operating anymore.

I know that for HA there is the need for shared storage (all nodes having acces to that shared storage). That can be a NAS or an independent node (with more slots for storage) running TrueNAS or something similar. Until i have that, i will try to have all my important services like Adguard, NPM, NextCloud, guac, passwd mng and etc on the first nodes (the ones with DRBD/ ZFS replication). On the third node i will keep my openvas lxc, kali linux vm, wireguard an other no so important to be always online services and i will rely on back-ups and manual restoration.

My question would be, can i somehow configure HA without shared storage?

Later edit: i saw something interesting about rsync through ssh through cron jobs, for syncing local back ups from one node to another (in case of a failed ssd).

 

[LnxBil]

My question would be, can i somehow configure HA without shared storage?

That really depends on the what you understand of HA or what you require. For me, this is in most cases equivalent to crash-consistent failover VMs. So no dataloss besides the crash itself. This only works with distributed shared storage (CEPH) or dedicated shared storage on a HA storage (e.g. dual controller).

HA can also mean start the VM on another node and don't care about the data consistency, because there is only a very, very low change of that. This may be the case for your setup. If you don't change much (or all the time of day), you may can tolerate data loss of a couple of minutes, so ZFS replication could be just fine.