Ceph very low performance 12MB/s

[felipemb]

Hi,

I need help with my clúster... I have the following scenario of cluster with 3 nodes and Ceph with 2 nodes:

Cluster



Ceph

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.0.20.11/24
     fsid = b9b27296-a27b-400a-9fcf-aebb4a767bc2
     mon_allow_pool_delete = true
     mon_host = 10.0.20.11 10.0.20.12 10.0.20.13
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 10.0.20.11/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.nodo1]
     public_addr = 10.0.20.11

[mon.nodo2]
     public_addr = 10.0.20.12

[mon.nodo3]
     public_addr = 10.0.20.13

My problem is that the performance is very low, here the results:




In some moments the speed increases up to 110MB/s, but I am doing the test with SSD and it should be higher than 350MB/s

Can someone help me finding a solution to improve performance?


Thank you!

 

[Klaus Steinberger]

-> CEPH with only 2 OSD Nodes should never be used for production !
for data security it is necessary to have a 3/2 replication rule! Read the CEPH docs!

-> What is your network speed? 110 MB/s fits to 1 GBit/s ?

 

[felipemb]

Klaus thanks for your answer...

1. It is not possible to have 3 nodes in the cluster and monitor and only 2 nodes replicating with Ceph? Or what recommendation can you give me to make a cluster with replication in 2 nodes? ZFS-replication?
2. I have created a round-robin bond with 2 network ports each with a speed of 1Gbps, in theory 250MB/s

Thanks for your time!

 

[Klaus Steinberger]

No only 2 nodes replicating with is unsafe, its is per design that you need at least 3 nodes
hardcore CEPH People even recommend more Nodes .... but 3 works well.

Also 1 G Links are way to slow for decent CEPH Performance. Remember CEPH has to replicate _any_ write to all 3 nodes before it can acknowledge the transfer !

For a working productions CEPH setup (besides experimental setups) you need at least nodes with 10 GBit/s links for CEPH Backend and 10 GBit/s links for Frontend (faster is better of course!) also you should have more than 2 OSD's per node!

 

[AlexLup]

If you get a third node it also speeds up that considerably. Also for small clusters increasing the PG count helps alot on the same principle as more nodes = more disks that can participate in satisfying the request..

 

[felipemb]

Thank you very much for clearing my doubts!