Hyper-Converged Ceph 3 node Cluster direct attach - Proxmox VE

[Caderize]

Dear all,

I'm asking if it is possibile to setup a 3 node Proxomox Cluster with Ceph where each node is direct attached, via 10 GBit Ethernet Link, to the other two nodes.
So each node has 2 cluster-only ethernet interfaces.

Summary (sorry for not providing an image):

Node A: 1st link connected to: Node B 1st link
Node A: 2nd link connected to: Node C 1st link
Node B: 1st link connected to: Node A 1st link
Node B: 2nd link connected to: Node C 2nd link
Node C: 1st link connected to: Node A 2nd link
Node C: 2nd link connected to: Node B 2nd link

It would be nice, but no so easy do implement, if also a failure of one link (ex. the link from "Noda A" to "Node B") will be caried by an "alternative" path,
so "Node A" sees "Node B" through "Node C".

Any suggestion will be apreciate.

Thanks

Caderize

 

[Mihai]

That's how we set up our 3 node corosync network.

Create a bond between the two interface on each node, with bond mode: Broadcast.

Node A: 1st + 2nd link = bond0 (broadcast mode)
Node B: 1st + 2nd link = bond0 (broadcast mode)
Node C: 1st + 2nd link = bond0 (broadcast mode)

Then set a static ip on each node for the bond connection. Obviously, each node should have a separate ip.

Here's an example:



This method will not allow you to see a node through another node. To do that you're essentially trying to turn each node into a network switch as well, I have no idea how to do that, but there's always a way with Linux probably.

 

[Digitaldaz]

I have a set up similar to this for about three years now without problem. I barely remember setting it up but I know the corosync was redundant in that we have a 1GB connection through a regular switch and then a couple of Intel X520-DA2 in each node directly attached.

 

[Alwin]

Have you seen the howto on our wiki yet?
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

 

[Caderize]

Thank you all!

I will use/follow: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server guide.

Thanks