CEPH performances

[eglyn]

Hi everyone

I have a 3 nodes cluster with 5 SSD 2To in each one.
I build a ceph storage with 15 OSD.
the cluster have a dedicated network 10Gb/s for this storage.

I benchmarked my storage pool with rados, but I don't know if the results are good or bad :/

Execute a write test for 10 seconds to the newly created storage pool:

Total time run:         11.2438
Total writes made:      1023
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     363.935
Stddev Bandwidth:       114.582
Max bandwidth (MB/sec): 456
Min bandwidth (MB/sec): 56
Average IOPS:           90
Stddev IOPS:            28.6455
Max IOPS:               114
Min IOPS:               14
Average Latency(s):     0.159838
Stddev Latency(s):      0.211775
Max latency(s):         3.51481
Min latency(s):         0.0251469

Execute a sequential read test for 10 seconds to the storage pool:

Total time run:       5.47196
Total reads made:     1023
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   747.812
Average IOPS:         186
Stddev IOPS:          186.741
Max IOPS:             360
Min IOPS:             0
Average Latency(s):   0.0539229
Max latency(s):       2.91111
Min latency(s):       0.0129373

Execute a random read test for 10 seconds to the storage pool:

Total time run:       12.2125
Total reads made:     2888
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   945.918
Average IOPS:         236
Stddev IOPS:          136.993
Max IOPS:             376
Min IOPS:             0
Average Latency(s):   0.057928
Max latency(s):       3.06877
Min latency(s):       0.00381988

Can someone tell me if this results are good or not ?

thx

 

[Zerstoiber]

you have network problems! this is where you need to take a look at first and foremost

Max latency(s): 3.51481
Max latency(s): 2.91111

 

[czechsys]

You can compare some things with
https://forum.proxmox.com/threads/p...-a-proxmox-ve-ceph-reef-cluster.137964/unread

 

[eglyn]

you have network problems! this is where you need to take a look at first and foremost

Max latency(s): 3.51481
Max latency(s): 2.91111

I think it was a bug, I launched several tests after, and latency seems normal

Total time run:         10.079
Total writes made:      1193
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     473.458
Stddev Bandwidth:       56.8835
Max bandwidth (MB/sec): 540
Min bandwidth (MB/sec): 340
Average IOPS:           118
Stddev IOPS:            14.2209
Max IOPS:               135
Min IOPS:               85
Average Latency(s):     0.134744
Stddev Latency(s):      0.0843132
Max latency(s):         0.52104
Min latency(s):         0.0226595

 

[jsterr]

Please post the exact commands that you are using for the benchmark. 10Gbit/s is often a bottleneck in flash-setups as you can only get 1GB/s from the network and ceph also writes 3 replicas, so you even have less then that. Its recommended to:

• use lacp with layer 3+4
• maybe even split ceph public and ceph cluster (but only when your on 10gbit-network)
• use jumbo-frames
• please also post the bandwith youll archieve with iperf to see if your network works and can reach 9-10Gbit/s

Please also post:

• the type of ssd (vendor + model) you are using

 

[eglyn]

Please post the exact commands that you are using for the benchmark. 10Gbit/s is often a bottleneck in flash-setups as you can only get 1GB/s from the network and ceph also writes 3 replicas, so you even have less then that. Its recommended to:

• use lacp with layer 3+4
• maybe even split ceph public and ceph cluster (but only when your on 10gbit-network)
• use jumbo-frames
• please also post the bandwith youll archieve with iperf to see if your network works and can reach 9-10Gbit/s

Please also post:

• the type of ssd (vendor + model) you are using

I used this commands:

• rados bench -p testbench 10 write --no-cleanup
• rados bench -p testbench 10 seq
• rados bench -p testbench 10 rand

Tests Iperf3:




Results of all tests:

Execute a write test for 10 seconds to the newly created storage pool:

Total time run:         10.079
Total writes made:      1193
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     473.458
Stddev Bandwidth:       56.8835
Max bandwidth (MB/sec): 540
Min bandwidth (MB/sec): 340
Average IOPS:           118
Stddev IOPS:            14.2209
Max IOPS:               135
Min IOPS:               85
Average Latency(s):     0.134744
Stddev Latency(s):      0.0843132
Max latency(s):         0.52104
Min latency(s):         0.0226595

Execute a sequential read test for 10 seconds to the storage pool:

Total time run:       3.55769
Total reads made:     1193
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1341.32
Average IOPS:         335
Stddev IOPS:          10.7858
Max IOPS:             361
Min IOPS:             341
Average Latency(s):   0.0455265
Max latency(s):       0.260207
Min latency(s):       0.0129035

Execute a random read test for 10 seconds to the storage pool:

Total time run:       10.0626
Total reads made:     3390
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1347.57
Average IOPS:         336
Stddev IOPS:          31.9592
Max IOPS:             381
Min IOPS:             274
Average Latency(s):   0.0467068
Max latency(s):       0.741604
Min latency(s):       0.00406465

For Jumbo Frame, I need to set MTU to 9000 on each network card, that's right ?

 

[jsterr]

Yes 9000 on all ceph network ports, but might result in short ceph hickup.

Please also post:

• the type of ssd (vendor + model) you are using

 

[sb-jw]

If switches are involved, they should first be configured for jumbo frames.

 

[eglyn]

Yes 9000 on all ceph network ports, but might result in short ceph hickup.

Please also post:

• the type of ssd (vendor + model) you are using

Ah yes, I forgot sorry, here the model:

Samsung SM883 - 1920 Go​