Ceph and ECC/Non-ECC nodes

[gasmanc]

So I’m trying to cobble together a homelab cluster. Its a mixed bag with an e3-1230v3 16gb ECC node, an intel nuc 32gb non-ecc (USB 3 hdd external enclosure) and soon an AMD 3700x with 32gb ECC ram.

How does Ceph play with such an asymmetric setup? Each node will have 3x3TB hdd and 10gbe (the nuc has a thunderbolt to 10gbe adapter). I’m slightly concerned about the nuc node with non-ecc ram. Is there any way to preference data coming into the cluster? I’d rather it hit the nodes with ECC and then get replicated to the non-ECC node. At least if there’s a bit flip in RAM it shouldn’t get distributed to the rest of the cluster. Can this be done?

 

[Alwin]

I'd argue that the USB connected disks will bring more trouble. And while you can prefer OSDs, the NUC will most likely face data repairs more often. Personally, I would not use Ceph in this setup. I'd go with ZFS on the e3 and AMD, then use the storage replication for redundancy. And best use disks directly connected to the SATA ports.
https://pve.proxmox.com/pve-docs/chapter-pvesr.html

 

[gasmanc]

Yeah, The USB-C enclosure isn’t an elegant solution, but a stop gap until I can get another node up. It’s a 4 disk enclosure that appears as single disks. Bandwidth is limited to 5gbps but I’m only running 3 3tb disks. I’m not expecting performance, but do want reliability. I’ve already got a raidz2 setup, but want to move to Ceph due to easier upgrades. For me to add another vdev now, that’s 6 disks.

Could you point me in the right direction on how to preference OSDs? I’m guessing it’s still no guarantee that data won’t hit the nuc OSDs first, right? Hoping to quickly add a proper third node and relegate the nuc to a Ceph mon.

Thanks for your help

 

[gasmanc]

Alternatively, I could just leave the one drive in the nuc connected to SATA. But that would leave an asymmetric OSD setup with 2 nodes having 3-4OSDs per node and the nuc only having a single OSD.

I think I’ll just have to bite the bullet. Trial it all and see if it bites me in the b**t.