5 Node Ceph Cluster without a switch

[Havok636]

Hello,

Is is possible to set up a viable 5 node ceph cluster without a switch? I have read the wiki and I understand that 3 nodes is ideal. I have 5 servers

Stack:
Dell R630: 2x1GBe, and 2x10GB SFP+ DAC
Dell R740XD: 2x1GBe, and 2x10GB SFP+ DAC
Dell R740XD: 2x1GBe, and 2x10GB SFP+ DAC
Dell R740XD: 2x1GBe, and 2x10GB SFP+ DAC
Dell R720: 2x1GBe, and 2x10GB SFP+ DAC

I have each server connected to my network via 1GBe. And all server connected in series down let line, as well as a connection from the 720 on the bottom to the 630 on top (loop).

I am looking to have the DAC lines separate for the network but as by dedicated data connection for CEPH. is this possible. If so, has an extreme hobbyist, would someone be willing to walk me through this porcess?

I have borked my network and setup so many times this weekend that I am a bit burnt out from resetting everything and re-installing proxmox on each node.

 

[readyspace]

Yes, you can run a 5‑node Ceph cluster with just DAC cables (Max number of nodes recommended for this setup will be 3 nodes) —if you treat the SFP+ links as a routed L3 ring or mesh, not a Layer‑2 loop. Proxmox has a documented method for this (called “Meshed Network for Ceph Servers”) that uses either routed with fallback or RSTP, and it scales better for small clusters like yours.

But its really easier just by getting a switch to build a 5 node cluster.

 

[aaron]

We have customers who do run 5-node full-mesh clusters, for example with 4x 25Gbit NICs.
Do not go for a ring topology as that could break in 2 places and then you have problems.

The Routed with Fallback method is what you want https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Routed_Setup_(with_Fallback)
And all nodes need to be connected to each other directly. The diagram, if you place the 5 nodes in a circle, will end up as a pentragram ;-)