Question about poor ceph perfomance

[katsil]

Good afternoon. I am experiencing problems with poor ceph performance, with dd showing good results, but integration with a VM with a disk on ceph raises doubts and problems. For example, the time echo command can be executed both 00.000 ms and 02.000ms and varies in this interval, that is, it can be very long.

My setup consists of 3 servers with a FRR network on a 10Ge fiber network. SSD 1Tb drives are used on all servers, they are the same. I also did some tests and they also vary - iops on osd.0-2 can be either 100 or 20 periodically. Help me find the cause and the problem and fix it, please.


ceph tell osd.* bench

osd.0: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 2.6786347930000001,
    "bytes_per_sec": 400854131.66661572,
    "iops": 95.571072498945171
}
osd.1: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 2.4059650779999999,
    "bytes_per_sec": 446283212.42823958,
    "iops": 106.40220938402166
}
osd.2: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 10.055356401999999,
    "bytes_per_sec": 106783069.74643226,
    "iops": 25.459067761047425
}

Some useful graphs - Grafana - 1. https://grafana.superhub.xyz/dashboard/snapshot/svaZCJ6lgCBxkqqaXqFfnrkyFOZZcvcE, 2. https://grafana.superhub.xyz/dashboard/snapshot/U19lOH1AtFnLnt82ytrhjuFOEODqPZcO


1008 bytes from 192.168.200.4: icmp_seq=1 ttl=64 time=0.098 ms
1008 bytes from 192.168.200.4: icmp_seq=2 ttl=64 time=0.100 ms
Iperf ~9.8G

 

[sb-jw]

Which SSDs do you use?

Basically, your expectations of a CEPH with 3 OSDs should be very cautious, but I find the values acceptable.