Hyper-Converged Ceph 3 node Cluster direct attach - Proxmox VE

Caderize

New Member
May 6, 2020
2
0
1
48
Dear all,

I'm asking if it is possibile to setup a 3 node Proxomox Cluster with Ceph where each node is direct attached, via 10 GBit Ethernet Link, to the other two nodes.
So each node has 2 cluster-only ethernet interfaces.

Summary (sorry for not providing an image):

Node A: 1st link connected to: Node B 1st link
Node A: 2nd link connected to: Node C 1st link
Node B: 1st link connected to: Node A 1st link
Node B: 2nd link connected to: Node C 2nd link
Node C: 1st link connected to: Node A 2nd link
Node C: 2nd link connected to: Node B 2nd link

It would be nice, but no so easy do implement, if also a failure of one link (ex. the link from "Noda A" to "Node B") will be caried by an "alternative" path,
so "Node A" sees "Node B" through "Node C".

Any suggestion will be apreciate.

Thanks

Caderize
 
That's how we set up our 3 node corosync network.

Create a bond between the two interface on each node, with bond mode: Broadcast.

Node A: 1st + 2nd link = bond0 (broadcast mode)
Node B: 1st + 2nd link = bond0 (broadcast mode)
Node C: 1st + 2nd link = bond0 (broadcast mode)

Then set a static ip on each node for the bond connection. Obviously, each node should have a separate ip.

Here's an example:

1588783628882.png

This method will not allow you to see a node through another node. To do that you're essentially trying to turn each node into a network switch as well, I have no idea how to do that, but there's always a way with Linux probably.
 
I have a set up similar to this for about three years now without problem. I barely remember setting it up but I know the corosync was redundant in that we have a 1GB connection through a regular switch and then a couple of Intel X520-DA2 in each node directly attached.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!