Slow performance in Ceph

[terramande]

hey,

i have set up a Cluster with 3 nodes, Its old hardware i know and i dont awaiting rocktspeed but a write sped in Ceph is a bit slow i thing,
i have: 1 - Node 40 x Intel(R) Xeon(R) CPU E5-2650L v2
2 - 16 x Intel(R) Xeon(R) CPU E5620
3 - 24 x Intel(R) Xeon(R) CPU L5640

10 osd hdd are in the hole system.

i only get very low performance with 2MiB/s

what aditinal info i can provide for help.

best regars

 

[gurubert]

What type of network do you have?
How does the RAM usage look like? Do the nodes need to swap?
Do you really have HDD-only OSDs or do you use SSDs for their RocksDB?

 

[terramande]

The 3 nodes have a gigabit networt to the rest of the net and on a diffrent ip range there are konectet via 10gb network, i tested both speeds with iperf3 and get nearly that speed.
Nr1 one has 100gb ram and the other both 40gb and max usag ist 40%
and yes i have only hdds for the OSDs

 

[gurubert]

The 3 nodes have a gigabit networt to the rest of the net and on a diffrent ip range there are konectet via 10gb network, i tested both speeds with iperf3 and get nearly that speed.

Does the Ceph cluster network utilize the 10G connection? And the public network is on the 1G connection?

i have only hdds for the OSDs

This would be your bottleneck.

Ceph uses random IO especially for the RocksDB (management part of the OSD). HDDs cannot cope with that very good due to their limited IOPS. In the end you get very poor performance.
A three node cluster is also not helpful as all the nodes are busy when writing data (three copies). More nodes would help.

As you only have 10 disks you may be better off with them in a single server, RAID6 on them and exporting them via iSCSI or NFS.

 

[terramande]

yes, i think so but how can i check that ceph uses the 10g? fot that reason i set the clusternetwork on the 10g ip range,

your sugestion is to use in all 3 node a ssd for RocksDB? Cehp uses the SSD by its own for RocksDB or i have to tell them?

 

[gurubert]

Please make yourself familiar with Ceph. IMHO it is not suitable for such a small setup.

 

[VictorSTS]

yes, i think so but how can i check that ceph uses the 10g? fot that reason i set the clusternetwork on the 10g ip range,

Ceph cluster network is used only by OSD's to create the replicas. All client access and daemon communication (Monitor, Manager, etc) go through the Ceph Public network.