CEPH setup
- Thread starter PmUserZFS
- Start date
Code:
Doesnt affect latency and QD1 heavy workloads.
Running 2x25GB for both public and cluster.
Connectx5 for public optimized for latency Intel XXV710 for clusterIt's not about high load or bandwidth — it's all about how quickly the network can return an Ack. I feel like this aspect is being completely overlooked here.
Last edited:
a ACK is small, the network isnt the limiting factor for acks, its ceph "slow" architecture.
Code:
[ CEPH CLIENT ]
|
| 1. Request Cluster Map (If needed)
v
[ CEPH MONITORS ]
|
| 2. Returns Cluster Map & CRUSH Map
v
[ CEPH CLIENT ]
|
| 3. Computes PG & OSD locations using CRUSH
| Object -> PG (Placement Group) -> OSDs [Primary, Secondary, Tertiary]
|
| 4. Direct Write (Front-End/Public Network)
v
+-----------------------------------------------------------------------------------+
| CEPH STORAGE CLUSTER |
| |
| +-----------------------+ |
| | PRIMARY OSD | <========== [ Acting Set for PG ] |
| +-----------------------+ |
| | | |
| | 5a. Replicate| 5b. Replicate (Back-End/Cluster Network) |
| v v |
| +-----------------------+ +-----------------------+ |
| | SECONDARY OSD | | TERTIARY OSD | |
| +-----------------------+ +-----------------------+ |
| | | | |
| | 6a. Commit | | 6b. Commit |
| v +-------+-----+ |
| +-----------------------+ | |
| | LOCAL DISK | | |
| +-----------------------+ | |
| | |
| +---------------------+ |
| | 7. Secondary & Tertiary Ack Replication Success |
| v |
| +-----------------------+ |
| | PRIMARY OSD | |
| +-----------------------+ |
| | |
| | 8. Commit to Local Disk & aggregate all Acks |
| v |
+--------|--------------------------------------------------------------------------+
|
| 9. Write Acknowledged (Success!)
v
[ CEPH CLIENT ]I thought that using ConnectX-5 for Ceph and Intel XXV710 for the public and cluster would simply speed things up without incurring additional costs, but I just realized that they’re using the 710 for Ceph—which requires low latency—and ConnectX-5 for parts where it isn’t necessary.
It’s part of the specification to wait for all synchronizations to complete before writing. I simply thought that if we could speed up the round-trip ACKs on the network connecting the OSDs, we should do so.
If you reject the specification, then Ceph is not an option.
It’s part of the specification to wait for all synchronizations to complete before writing. I simply thought that if we could speed up the round-trip ACKs on the network connecting the OSDs, we should do so.
If you reject the specification, then Ceph is not an option.
I have to benchmark so see if the x5 have lower latency then xxv710. My initial thinking was only for VMs to have fast access.
Yes ceph is for ha,data integrety and scale.
Interesting, since when you just look at packets on the the network and latency, I wonder hos chatty eg, pps ceph is. The packets could add up so that very small difference makes a difference.
I swapped the nics and it made a huge difference, I know mellanox is better, but damn.
Ceph Network Swapping Test Summary
| Workload Type | Metric | Before (Old Public/Cluster Setup) | After (ConnectX-5 for Cluster Network) | Performance Impact |
|---|---|---|---|---|
| Baseline 4K Writes (Continuous Background) | IOPS Avg Latency Max Latency | 463 IOPS 2.13 ms 15.93 ms | 468 IOPS 2.11 ms 10.43 ms | Minor IOPS gain Identical avg latency 34% lower latency spikes |
| Burst 64K Writes (5s Heavy VM Write) | Throughput IOPS Avg Latency Max Latency | 161 MiB/s 2,577 IOPS 6.19 ms 34.05 ms | 165 MiB/s 2,633 IOPS 6.05 ms 13.63 ms | 2.5% bandwidth increase 2.1% IOPS increase Faster write replication 60% lower latency spikes |
| Burst 128K Reads (10s Heavy VM Read) | Throughput IOPS Avg Latency | 4,037 MiB/s 32,442 IOPS 987.9 µs | 4,231 MiB/s 33,850 IOPS 942.1 µs | 4.8% bandwidth increase 4.3% IOPS increase Slashed client-side read latency |
Key Takeaways
- Eliminated Micro-Stuttering: The most massive improvement is the 60% drop in maximum write latency (from 34.05 ms down to 13.63 ms). The faster ConnectX-5 pipeline prevents OSD replication queues from backing up during sudden VM bursts.
- Higher Client Performance: Moving client traffic to the dedicated Intel XXV710 network allowed cache read bursts to scale up, successfully breaking past 4.2 GiB/s with lower access latencies.
I'm glad it's been fixed. It would be a waste if it weren't installed in the right place.
*I think even the ConnectX-4 outperforms the XXV710.
You should be grateful for @MarkusKo's advice.
*If we don't need 50Gbps bandwidth, wouldn't a single 25G x 1 be faster for lower latency and better ACK times than 25G x 2 bonding?
*I think even the ConnectX-4 outperforms the XXV710.
You should be grateful for @MarkusKo's advice.
*If we don't need 50Gbps bandwidth, wouldn't a single 25G x 1 be faster for lower latency and better ACK times than 25G x 2 bonding?
Last edited:
No, the hashing is very fast. The failover can be slower compared to lvl7 detection though.