Recommended Max Ceph Disks / Nodes
- Thread starter starnetwork
- Start date
see:
http://docs.ceph.com/docs/jewel/start/hardware-recommendations/
ceph doc said something like
-2 osd/ 1core+hyperthreading.
- 2gb ram by osd
better to have 1 disk controller for 8 disks. (avoid oversubscribing dataplane)
then if you need fast latency/ a lot of small random iops, try to have faster frequencies cpu.
http://docs.ceph.com/docs/jewel/start/hardware-recommendations/
ceph doc said something like
-2 osd/ 1core+hyperthreading.
- 2gb ram by osd
better to have 1 disk controller for 8 disks. (avoid oversubscribing dataplane)
then if you need fast latency/ a lot of small random iops, try to have faster frequencies cpu.
Hi,
this Calculations are "Per Node", and it's OK
but I mean how many nodes on 10GB network it's recommended to do, since there is lot of data sync between nodes, no?
this Calculations are "Per Node", and it's OK
but I mean how many nodes on 10GB network it's recommended to do, since there is lot of data sync between nodes, no?
For ceph this means, the more the better. As you distribute the cluster traffic among servers and spread out your disks that are taking care of read/writes objects for a certain PG. The good part with ceph is, that you can start small and grow as needed (performance & capacity).
I recommend 10 gbit for the beginning. Not only for bandwith, but latency, which also matter. And (optical!) 10gbit have much less latency than copper 1 gbit.
For better latency you can also check out the direct attached copper cables (DAC), IMHO they are better to handle then optical.
For better latency you can also check out the direct attached copper cables (DAC), IMHO they are better to handle then optical.
Hi,
dac cable also using SFP so the latency almost the same. It's basically two sfp module connected with coax. But there is a regular RJ45 copper based 10gbit, where latency also wrong. This is because of interface, not because of the material (rj45 vs sfp+)
Last edited:
I have read somewhere quite some time ago that having 1 phisical CPU with high clock is better than having 2 CPU for ceph, is this still that case ?
Also how would you know if it is time to upgrade your network from 10Gb to something faster- we are maxing out at about 1.5 Gbps when looking at the network live (the ceph private and public interface). Also our disk controller is has only 12 Gbps throughput so I assume if we were to upgrade the network we would have to upgrade the controllers as well and make sure that the disks' speed when added is fast enough to use that speed (of the network and new hardware controlers).
Also how would you know if it is time to upgrade your network from 10Gb to something faster- we are maxing out at about 1.5 Gbps when looking at the network live (the ceph private and public interface). Also our disk controller is has only 12 Gbps throughput so I assume if we were to upgrade the network we would have to upgrade the controllers as well and make sure that the disks' speed when added is fast enough to use that speed (of the network and new hardware controlers).
yes, for low iodepth / small block workload, better to have big frequencies to speedup thing.
if you are only doing big block workload, don't matter too much.
Its really depending on your usa case and your hardware. For example SSD only CEPH with lot of small transaction require more CPU than spinner disks with lot of huge block operationals. And also, you want hyper converged ceph or regular.
Eq. if you have a 12 bay server with 10 regular sata HDD (+2 hdd for the OS), theoretically you not need more than 10gbit network (if we calculating 100 mbit /s avarage speed per sata disk.) But you can find examples and calculating methods on ceph documentation and mailling lists.