Adding a 4th ceph node with old cpu

[margerko]

Hello!

Context / current setup

• 3 nodes, each with 2× enterprise U.2 NVMe (PCIe 4.0 x4) OSDs
• Network: 2×25 GbE per node
• Replicated pools (size=3)
• Planning to add a 4th node with the same NVMe drives, but the motherboard only supports PCIe Gen3, and the CPU is Xeon D-1541 (older/weaker)

Questions

1. Will this mixed node (Gen3 bus + weaker CPU) drag down overall cluster throughput/latency?
2. Where would I notice it most (writes vs reads, recovery/rebalance)? Most of volume is 1mb blocks

What I’m thinking so far

• With 4 hosts and replication=3, roughly 75% of PGs will include the slower host at least once. For writes, latency is gated by the slowest replica in the acting set, so some slowdown seems inevitable.
• The Gen3 vs Gen4 gap matters mostly for big sequential I/O; with 2×25 GbE and Ceph’s small I/O patterns, the CPU (D-1541) is likely the bigger limiter than the PCIe link.

Would love real-world experiences from anyone who mixed Gen3/Gen4 NVMe and older CPUs in Ceph: how big was the hit, and which mitigations worked best?