3node brand new ceph cluster vs 5node mixed ceph cluster

[silvered.dragon]

I have an old 3node ceph cluster with HP gen8 servers, 2x Xeon E5-2680 @ 2.70GHz (turbo 3,50 GHz) 16core and 64GB DDR3 RAM for each node.
We bought some almost new HP gen10 servers with 2x Gold Xeon 6138 @ 2.0GHz (turbo 3,70 GHz) and 128GB DDR4 RAM for each node.
So there is a huge jump in terms of performance from the old setup to the new one.
Considering that:

• All servers have the same amount of HPE HDD 15K 600GB(but obviously the gen8 servers have old used HDDs, the new gen10 ones have zero hours HDDs).
• Net interfaces are the same for all the servers with multiple 10Gbs SFP+.

At the beginning I was planning to simply replace the old 3node ceph cluster with the new one, and use the old servers for other non intensive tasks, like file servers or a proxmox backup server or weathever in a ZFS config, but now I'm thinking to create a 5node cluster mixing the old ones with the new ones, and use the old ones just for ceph so no intensive VMs or CTN tasks, probably I will put some realy light CTN for file shares or various light utilities.

So my questions are:

• will the performance be impacted by this kind of setup?
• will the older HDDs impact the performance of the new ones?
• so is better 3 new nodes or 5 mixed nodes?

 

[jdancer]

Ceph really wants identical hardware to equally spread the load.


Can do mixed but the slowest machine will be the bottleneck.

 

[silvered.dragon]

Ceph really wants identical hardware to equally spread the load.


Can do mixed but the slowest machine will be the bottleneck.

Ok, but let's say you have a 50 node cluster and you want to a improve a little the performance, do I have to replace all 50 nodes?
Or on the countrary let's say one server is dead and I replace it with an older one, will the performance impact all the remaining 49?

 

[czechsys]

It depends. Everytime you will need something from slower node, your faster node will wait. On smaller cluster it will be more visible than on bigger.

Anyway, using 15k 600GBs disks is performance hit anyway for ceph no matter if they are old or new - performance is the same usually if on the same SAS/SATA protocol.

 

[silvered.dragon]

It depends. Everytime you will need something from slower node, your faster node will wait. On smaller cluster it will be more visible than on bigger.

Anyway, using 15k 600GBs disks is performance hit anyway for ceph no matter if they are old or new - performance is the same usually if on the same SAS/SATA protocol.

Can you clarify a little more

performance hit anyway for ceph

I used this setup for so much years in a heavy workload, I mean terabyte databases, webservers with 20k visits per weeks and so on, without any particular feeling of slowness, I understand that with SSDs I can push 100 times faster, but this doesn't means that I ever felt performance issues in my setup, so when you say perfomance hits anyway, you mean that with HDDs ceph will not impact too much CPU so gen 7 or gen 10 servers are the same?