Is a 3-node Full Mesh Setup For Ceph and Corosync Good or Bad

[x509]

Hello PROXMOX forum!

I am working on an infrastructure setup for a small-medium business. We are confused about the networking topology and my research hasn't given me a great answer. We are looking at a 3-node setup with mesh networking for Ceph (internal and public networks) and corosync. We mostly want to go the mesh route so that we don't have to buy 25Gb switches. I have read the PROXMOX docs on mesh networking, hyperconverged, etc. but the implementation details and real-world data seems to be lacking.

Our current hardware plan:

• Dell R7xx hosts.
• Each with a single ~20-core CPU.
• Each with 128GB RAM.
• Each with 5x 1.92TB SSDs.
• Each with 4x 25Gb ports.
• Each with 4x 10Gb ports (Base-T, ethernet).
• Each with 2x 1Gb ports.

The plan was to use:

• 2x 25Gb ports for Ceph internal traffic (DAC Cables, Mesh: node A->B, B->C, C->A).
• 2x 25Gb ports for Ceph public traffic (cabled as above). Possibly also VM migration traffic.
• 2x 10Gb for VM traffic to LAN.
• 2x 10Gb for management traffic (also for corosync backup).
• 2x 1Gb for corosync (Mesh: node A->B, B->C, C->A).

We have talked to a few PROXMOX partners about what we want to do. One is advocating strictly against a mesh network topology. Another says mesh networking is fine. When I asked about why the one partner is against mesh, they said it is due to the PROXMOX node having to handling networking duties and issues occurring when the node goes down. They basically said mesh is one of the worst things you can do and it is only for homelab situations.

Can anyone help me with some real-world implementation details/results and thoughts?

Further info: We do not expect to outgrow a 3-node cluster, plus we can always upgrade these nodes since we are buying pretty low specced hardware. Our current infrastructure is a 2-node vSphere cluster running StarWind VSAN with 2x 25Gb ports direct connected between nodes. CPU, RAM, and storage is nearly the same as noted above, just jammed into 2 nodes instead of 3. About 25 VMs: file servers, domain controllers, database servers, application servers, web servers. CPU is typically <10%, RAM usage is typically ~50%, storage is ~50% used, networking utilization is low.

 

[SteveITS]

Have a read through https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/. One challenge with 3 nodes and 3/2 replication is that if you reboot or shut down one node you’re already at the minimum.

Proxmox documents mesh : https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server. I’ve not used it though.

 

[bbgeek17]

@x509 , welcome to the forum.

We don’t use Ceph or mesh networking in our deployments. That said, if you plan to engage a Proxmox Partner, I would recommend following their guidance, they will ultimately be responsible for supporting your infrastructure.

If this is a production environment for your business and you don’t already have hands-on experience with mesh networking, it would be easier to invest in a simple FS switch (ideally two). This allows you to focus on running the business rather than troubleshooting a complex network design.

And, if you are providing consultancy services to a customer, also use dedicated switches. It simplifies the architecture and puts you in a much stronger position to support and troubleshoot the environment effectively.

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Johannes S]

One challenge with 3 nodes and 3/2 replication is that if you reboot or shut down one node you’re already at the minimum.

This has nothing to do with the pros and cons of a mesh network though. And there are still usecases, where three nodes are ok, because they can tolerate the loss of one node and consider the failure of a second node a tolerable risk.

 

[SteveITS]

nothing to do with the pros and cons of a mesh network

It does not, but OP has 3 nodes and does not intend for more. Yes it will work with 3.

 

[bl1mp]

Hi x509,
we used it in my former company and it worked flawless, but there are some points in sizing and handling you should be aware of.

In this setup, I would definitely emphasize to include monitoring and backup, because you run with the minimum level of redundancy. (As you always should)

If you already in contact with a Proxmox Partner ask them to explain, which potential issues they see in which meshed network setup.

We used the broadcast setup. This is not providing the best bandwith, but easy to configure and has low entrance barrier for Linux networking.

BR, Lucas