CEPH setup
- Thread starter PmUserZFS
- Start date
Code:
Doesnt affect latency and QD1 heavy workloads.
Running 2x25GB for both public and cluster.
Connectx5 for public optimized for latency Intel XXV710 for clusterIt's not about high load or bandwidth — it's all about how quickly the network can return an Ack. I feel like this aspect is being completely overlooked here.
Last edited:
a ACK is small, the network isnt the limiting factor for acks, its ceph "slow" architecture.
Code:
[ CEPH CLIENT ]
|
| 1. Request Cluster Map (If needed)
v
[ CEPH MONITORS ]
|
| 2. Returns Cluster Map & CRUSH Map
v
[ CEPH CLIENT ]
|
| 3. Computes PG & OSD locations using CRUSH
| Object -> PG (Placement Group) -> OSDs [Primary, Secondary, Tertiary]
|
| 4. Direct Write (Front-End/Public Network)
v
+-----------------------------------------------------------------------------------+
| CEPH STORAGE CLUSTER |
| |
| +-----------------------+ |
| | PRIMARY OSD | <========== [ Acting Set for PG ] |
| +-----------------------+ |
| | | |
| | 5a. Replicate| 5b. Replicate (Back-End/Cluster Network) |
| v v |
| +-----------------------+ +-----------------------+ |
| | SECONDARY OSD | | TERTIARY OSD | |
| +-----------------------+ +-----------------------+ |
| | | | |
| | 6a. Commit | | 6b. Commit |
| v +-------+-----+ |
| +-----------------------+ | |
| | LOCAL DISK | | |
| +-----------------------+ | |
| | |
| +---------------------+ |
| | 7. Secondary & Tertiary Ack Replication Success |
| v |
| +-----------------------+ |
| | PRIMARY OSD | |
| +-----------------------+ |
| | |
| | 8. Commit to Local Disk & aggregate all Acks |
| v |
+--------|--------------------------------------------------------------------------+
|
| 9. Write Acknowledged (Success!)
v
[ CEPH CLIENT ]I thought that using ConnectX-5 for Ceph and Intel XXV710 for the public and cluster would simply speed things up without incurring additional costs, but I just realized that they’re using the 710 for Ceph—which requires low latency—and ConnectX-5 for parts where it isn’t necessary.
It’s part of the specification to wait for all synchronizations to complete before writing. I simply thought that if we could speed up the round-trip ACKs on the network connecting the OSDs, we should do so.
If you reject the specification, then Ceph is not an option.
It’s part of the specification to wait for all synchronizations to complete before writing. I simply thought that if we could speed up the round-trip ACKs on the network connecting the OSDs, we should do so.
If you reject the specification, then Ceph is not an option.