CEPH: public and cluster network

N

nl_alex

New Member
Nov 1, 2025
1
0
1
Hello all,

I am adding Ceph to an 3 node cluster. On each machine, I have one 10GbE and one 1GbE link available. What would be the better way of configuring my network?
- 10 GbE Public network, 1 GbE Cluster network
- 10 GbE Cluster network, 1 GbE Public network
- 10 GbE both networks

Thank you in advance for your opinions,

Alex
 
Last edited:
Did you already read the relevant parts of the manual?
https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network
https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

In general it's recommended to have one dedicated prinary network for Ceph and corosync and outside communication plus additional redundancy.

So:
- Network a corosync ( doesn't need to be fast 1gbit/s is enough)
- network b ceph and corosync redubdancy( fast 10 gbit/s or more )
- network c outside communication / corosync redundancy

With your limited options I would do this:

10 Gbit/s Ceph plus corosync primary
1 gbit/s outside communication/corosync redundancy

But depending on your needs, Budget and Hardware you might come to different conclusions. So: Read the documentation and Udos writeup:
UdoB

[TUTORIAL] Thread 'FabU: can I use Ceph in a _very_ small cluster?'

Ceph is great, but it needs some resources above the theoretical minimum to work reliably. My assumptions for the following text:
  • you want to use Ceph because... why not?
  • you want to use High Availability - which requires Shared Storage (note that a complete solution needs more things like a redundant network stack and power supplies)
  • you want to start as small (and cheap) as possible, because this is... “only” a Homelab
You plan for three Nodes. Each node has s single dedicated disk for use as an “OSD”. This is the documented...
  • Like
 
  • Like
Reactions: weehooey-bh and UdoB
You must log in or register to reply here.
Top Bottom