CEPH setup
- Thread starter PmUserZFS
- Start date
Code:
Doesnt affect latency and QD1 heavy workloads.
Running 2x25GB for both public and cluster.
Connectx5 for public optimized for latency Intel XXV710 for clusterIt's not about high load or bandwidth — it's all about how quickly the network can return an Ack. I feel like this aspect is being completely overlooked here.
Last edited:
a ACK is small, the network isnt the limiting factor for acks, its ceph "slow" architecture.
Code:
[ CEPH CLIENT ]
|
| 1. Request Cluster Map (If needed)
v
[ CEPH MONITORS ]
|
| 2. Returns Cluster Map & CRUSH Map
v
[ CEPH CLIENT ]
|
| 3. Computes PG & OSD locations using CRUSH
| Object -> PG (Placement Group) -> OSDs [Primary, Secondary, Tertiary]
|
| 4. Direct Write (Front-End/Public Network)
v
+-----------------------------------------------------------------------------------+
| CEPH STORAGE CLUSTER |
| |
| +-----------------------+ |
| | PRIMARY OSD | <========== [ Acting Set for PG ] |
| +-----------------------+ |
| | | |
| | 5a. Replicate| 5b. Replicate (Back-End/Cluster Network) |
| v v |
| +-----------------------+ +-----------------------+ |
| | SECONDARY OSD | | TERTIARY OSD | |
| +-----------------------+ +-----------------------+ |
| | | | |
| | 6a. Commit | | 6b. Commit |
| v +-------+-----+ |
| +-----------------------+ | |
| | LOCAL DISK | | |
| +-----------------------+ | |
| | |
| +---------------------+ |
| | 7. Secondary & Tertiary Ack Replication Success |
| v |
| +-----------------------+ |
| | PRIMARY OSD | |
| +-----------------------+ |
| | |
| | 8. Commit to Local Disk & aggregate all Acks |
| v |
+--------|--------------------------------------------------------------------------+
|
| 9. Write Acknowledged (Success!)
v
[ CEPH CLIENT ]