Testing CEPH IO Speed

[manitc]

My setup is 3 dell R820, H710 Raid0, 2x1TB SAS Raid1 for ProxMox, 6X1 TB SAS drives for CEPH, 10Gbe Network.

First Test, LXC stored on the local drive I get ~60 MB/S.
Second Test, LXC stored on the Ceph pool I get ~240MB/s.

Something is not adding up here. I would expect the CEPH pool to be significantly slower than the local storage.

Here are my results:

dd if=/dev/zero of=/tmp/test.dat count=1M bs=1024
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.45382 s, 241 MB/s

dd of=/dev/zero if=/tmp/test.dat count=1M bs=1024
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.05449 s, 523 MB/s

What's the best way to properly test DISK IO speed?

Thanks.

 

[gurubert]

With only 1GB test data you run the test on the memory cache.

https://yourcmc.ru/wiki/Ceph_performance has a nice writeup on IO benchmarking Ceph. Do not get distracted by his dismissal of Ceph performance trying to sell his own solution.

 

[manitc]

Thanks @gurubert, now I'm seeing the expected results.

 

[liberodark]

Hi,

On my side im use :

dd if=/dev/zero of=./perf-test bs=4k iflag=fullblock,count_bytes count=4G
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --bs=4k --iodepth=64 --readwrite=randrw --rwmixread=75 --size=4G --filename=/root/perf-test

Best Regards