CEPH Benchmark with 6 HDD WesternDigital 2TB

[unixpert]

Hi all,

I have 3 nodes CEPH Cluster with version 4.4.49-1-pve. All vm are very slow. Can you help me find the cause?

Network:
- Switch 10GB
- Interface 10GB Port

For every Server:
- OS System: 2 x 500GB raid 1
- OSD: 2 x 2TB HDD 7200rpm

CEPH Config:

[global]
     auth client required = cephx
     auth cluster required = cephx
     auth service required = cephx
     cluster network = 192.168.2.0/24
     filestore xattr use omap = true
     fsid = ece68812-200f-4b8e-9338-5ae8097f75e2
     keyring = /etc/pve/priv/$cluster.$name.keyring
     osd journal size = 5120
     osd pool default min size = 1
     public network = 192.168.2.0/24

[osd]
     keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.1]
     host = host002
     mon addr = 192.168.2.123:6789

[mon.0]
     host = host001
     mon addr = 192.168.2.121:6789

[mon.2]
     host = host003
     mon addr = 192.168.2.124:6789

This is my benchmark

root@host001:~# rados -p test3 bench 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_host001_10988
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        61        45   179.993       180    0.169214    0.303209
    2      16       102        86    171.98       164    0.240787    0.312535
    3      16       118       102   135.985        64      1.2364    0.380896
    4      16       126       110   109.988        32    0.977343     0.43432
    5      16       139       123   98.3891        52    0.257909    0.498998
    6      16       152       136   90.6566        52    0.944754    0.606939
    7      16       162       146   83.4194        40     1.48069    0.661344
    8      16       183       167   83.4906        84    0.390088    0.746517
    9      16       195       179   79.5466        48    0.601118    0.722079
   10      16       212       196   78.3912        68    0.114451    0.757158
Total time run:         10.649462
Total writes made:      213
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     80.004
Stddev Bandwidth:       51.5778
Max bandwidth (MB/sec): 180
Min bandwidth (MB/sec): 32
Average IOPS:           20
Stddev IOPS:            12
Max IOPS:               45
Min IOPS:               8
Average Latency(s):     0.789955
Stddev Latency(s):      0.62297
Max latency(s):         2.58016
Min latency(s):         0.084889

If i test with dd on vm, the speed is much slower.

[root@vm ~]# dd if=/dev/zero of=here bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 91.7209 s, 11.7 MB/s

With these hardware, can i get bandwidth around 200MB/s per VM ?

Thank you very much !

 

[dietmar]

With these hardware, can i get bandwidth around 200MB/s per VM ?

I guess not - spinning disk are much too slow for that (ceph needs more IOPS).

 

[unixpert]

I guess not - spinning disk are much too slow for that (ceph needs more IOPS).

Hi, Thanks. That means do I need to replace the SSD for OSD ? So, is my network model not the cause ?

 

[tschanness]

Either more OSDs or faster OSDs (or both). You did not show your network config. Is ceph on an extra 10G network?

 

[unixpert]

Either more OSDs or faster OSDs (or both). You did not show your network config. Is ceph on an extra 10G network?

Yes, i will add more OSDs or change to SSD. I use 10G network local for CEPH.

This is my interface local for every server.

auto eth7
iface eth7 inet static
        address 192.168.2.121
        netmask 255.255.255.0

Do I need to adjust mtu in this configuration?

 

[udo]

Hi all,
...
For every Server:
- OS System: 2 x 500GB WesternDigital raid 1
- OSD: 2 x 2TB WesternDigital
...

root@host001:~# rados -p test3 bench 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_host001_10988
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        61        45   179.993       180    0.169214    0.303209
    2      16       102        86    171.98       164    0.240787    0.312535
    3      16       118       102   135.985        64      1.2364    0.380896
    4      16       126       110   109.988        32    0.977343     0.43432
    5      16       139       123   98.3891        52    0.257909    0.498998
    6      16       152       136   90.6566        52    0.944754    0.606939
    7      16       162       146   83.4194        40     1.48069    0.661344
    8      16       183       167   83.4906        84    0.390088    0.746517
    9      16       195       179   79.5466        48    0.601118    0.722079
   10      16       212       196   78.3912        68    0.114451    0.757158
...

...

Hi,
you don't wrote which kind of hdd do you have exactly, but the benchmark values (normaly 10 second are not enough) shows good Values at the beginning, which drops rapidly...
This looks like an slow hdd (5400 rpm?), which is speed up with an bigger cache?! During cache fill, the speed is ok and drop then, if the cache is full.

I have an similiar config as test-cluser but with 1GB-Nics only.
The perfdata with six old 2TB-disks (7200, 3.5") are a little bit better (but yes, it's slow!)

rados -p test bench 60 write --no-cleanup  
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve03_874664
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        28        12   47.9907        48    0.939834     0.75636
    2      16        45        29   57.9875        68    0.843092    0.777084
    3      16        61        45   59.9885        64     1.07014    0.834778
    4      16        85        69    68.987        96    0.795774    0.815854
    5      16       105        89   71.1871        80    0.483717    0.808871
    6      16       127       111   73.9865        88     1.58905    0.803544
    7      16       147       131   74.8434        80    0.275539    0.793226
    8      16       165       149   74.4864        72    0.338184    0.795032
    9      16       187       171   75.9863        88      1.2687    0.810896
   10      16       208       192   76.7861        84    0.854789    0.803096
   11      16       222       206   74.8955        56    0.995979    0.812649
   12      16       244       228   75.9863        88    0.868137    0.807923
   13      16       264       248   76.2939        80     1.11455    0.811636
   14      16       284       268   76.5578        80    0.440731    0.808114
   15      16       306       290   77.3197        88    0.777698    0.807993
   16      16       325       309   77.2364        76     1.03011    0.804529
   17      16       344       328    77.163        76    0.612327    0.806375
   18      16       362       346   76.8756        72    0.865825    0.806239
   19      16       384       368   77.4604        88    0.594513    0.807383
...
   57      16      1130      1114   78.1638        64    0.599791    0.811475
   58      16      1148      1132   78.0573        72    0.601022    0.813862
   59      16      1168      1152     78.09        80    0.708484    0.813027
   60      16      1190      1174    78.255        88    0.395487    0.811647
Total time run:         60.664196
Total writes made:      1191
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     78.5307
Stddev Bandwidth:       11.2745
Max bandwidth (MB/sec): 96
Min bandwidth (MB/sec): 48
Average IOPS:           19
Stddev IOPS:            2
Max IOPS:               24
Min IOPS:               12
Average Latency(s):     0.814786
Stddev Latency(s):      0.368249
Max latency(s):         2.46571
Min latency(s):         0.254479

Udo

 

[unixpert]

hi, this is my disk:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Re
Device Model:     WDC WD2000FYYZ-01UL1B0
Serial Number:    WD-WCC1P0518416
LU WWN Device Id: 5 0014ee 25dfed953
Firmware Version: 01.01K02
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Jul 31 18:20:49 2018 ICT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Before, I ran the 1G switch and the same results as you. About 70-80MB/s.

After that, i run switch 10G with interface 10G network, the results are also not better.

Maybe my HDD is too slow.

Thanks so much!

 

[AlexLup]

Try a cache tier: http://technik.blogs.nde.ag/2017/07/14/ceph-caching-for-image-pools/
And lookup bcache, havent tested yet but forum posts on here say its really nice!