node cluster config

[dbrega]

I would like to create a cluster of three nodes with Ceph installed on both.
I would like to avoid using two switches because I don't need them, as the nodes are close to each other and are equipped with multiple 2.5GB network cards.
I was thinking of a configuration like the one shown in the image below.

I would like to know if anyone has any doubts about the correct functioning of this configuration.
Thank you in advance.

 

[UdoB]

I would like to know if anyone has any doubts about the correct functioning of this configuration.

Depending on your specific expectation... it will probably work. There is no magic involved

As soon as you need to tap in into one of the point-to-point networks you'll see the problem: you can not connect a fourth device, e.g. for trouble shooting.

2.5 GBit/s switches are cheap. Personally I would just buy one more. Instead of using the multiple NICs your way I would highly prefer to setup a separate connection for Corosync, a separate LAN for storage and a separate network for some (tagged only) VLANs. All of them using a switch, allowing for a fourth/fifth node to be attached next year...


Slightly off-topic in regard of your question: personally I would never run a three node Ceph installation in a productive environment. As soon as anything fails you are degraded - and you stay degraded as there is no redundancy to kick in and trigger self-healing - which is one of the killer-features Ceph offers.

I know several users do successfully use this construct, and of course: ymmv.

My own findings after a year of Ceph with 2.5 GBit/s and Mini-PCs: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

 

[dbrega]

Depending on your specific expectation... it will probably work. There is no magic involved

As soon as you need to tap in into one of the point-to-point networks you'll see the problem: you can not connect a fourth device, e.g. for trouble shooting.

2.5 GBit/s switches are cheap. Personally I would just buy one more. Instead of using the multiple NICs your way I would highly prefer to setup a separate connection for Corosync, a separate LAN for storage and a separate network for some (tagged only) VLANs. All of them using a switch, allowing for a fourth/fifth node to be attached next year...


Slightly off-topic in regard of your question: personally I would never run a three node Ceph installation in a productive environment. As soon as anything fails you are degraded - and you stay degraded as there is no redundancy to kick in and trigger self-healing - which is one of the killer-features Ceph offers.

I know several users do successfully use this construct, and of course: ymmv.

My own findings after a year of Ceph with 2.5 GBit/s and Mini-PCs: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

Why I will not connect the fourth device?
Is not possibile insert the fourt device in future?
The red connection network are configured for ceph public network to bond0 configured in balance-alb
The blue connection network are configured for ceph cluster network to bond1 configured in balance-alb
The black connection network are configured for proxmox cluster with only one network card for each nodes
If I want to add the fourth node in future, I think that it is possible to add in the ring of ceph public and cluster network, or not possible?

 

[UdoB]

Why I will not connect the fourth device?
Is not possibile insert the fourt device in future?

If I want to add the fourth node in future, I think that it is possible to add in the ring of ceph public and cluster network, or not possible?

What I see is a mesh: every node has direct contact to each other nodes. To add a fourth node every node needs an additional (third) NIC (or bond) dedicated exclusively for this.

If you configure it as a ring instead (which does work) then yes. Your may build a ring with ten nodes, if you wish to. Every bit to reach a node on "the other side of the ring" needs to go through each intermediate node.

Using a switch (or two, for redundancy) looks easier to setup, to maintain, to expand and to debug to me


Disclaimer: I have zero experience with this - everything I say in this specific context is based on my background knowledge, not on experience.