Proxmox and Ceph cluster with 5 nodes full mesh network.

[Gilberto Ferreira]

Hi there.
I want to create a full mesh network cluster with 5 nodes.
So, I went ahead and found this tutorial, which is for 3 nodes but I intend to adapt to work with 5 node. (1 of them, the 5th node will be only for quorum!):

https://packetpushers.net/blog/proxmox-ceph-full-mesh-hci-cluster-w-dynamic-routing/

So my question is: it's worth it? Anybody already give it a try?

Thanks a lot.

 

[Lukas Moravek]

Hello @Gilberto Ferreira

For mesh is recommended 3 node cluster, if you have more nodes or plan to expand, go with switches.

https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

L

Edit: correct link

 

[Gilberto Ferreira]

Hello @Gilberto Ferreira

For mesh is recommended 3 node cluster, if you have more nodes or plan to expand, go with switches.

https://forum.proxmox.com/threads/faster-failover-possible.36894/

L.

Thank for your reply.
Unfortunately my customer has 4 server and since ceph cluster is better in odd servers, we figured out that add a low profile machine as a 5th node will be a solution.
But he can afford to have a good switch for do a traditional network.

 

[aaron]

We have some customers who set up a 5-node full mesh with the OpenFabric variant by using 4-port 25 Gbit NICs. But those were all full Ceph nodes AFAIR. Seems to be working quite well for them and up to 5 nodes is still manageable regarding the wiring.

 

[Gilberto Ferreira]

We have some customers who set up a 5-node full mesh with the OpenFabric variant by using 4-port 25 Gbit NICs. But those were all full Ceph nodes AFAIR. Seems to be working quite well for them and up to 5 nodes is still manageable regarding the wiring.

Hi... Thank you for your considerations. The article that I sent in the opening thread, mention something about use the loopback interface, IPv6 and OSPF. Should this set up work out of the box?

Thanks

 

[aaron]

I assume it will work, but haven't tested this myself!
Looking at the date of the article and the history of our full-mesh article, it appears that it was written just a few months before we added the "routed with fallback" option, which is based on OpenFabric.

 

[Gilberto Ferreira]

I assume it will work, but haven't tested this myself!
Looking at the date of the article and the history of our full-mesh article, it appears that it was written just a few months before we added the "routed with fallback" option, which is based on OpenFabric.

I will check out both!
Thanks.