Ceph - writethrough caching method

[chrispage1]

I've been testing various disk caches with Ceph RBD. I'm using the writethrough cache mode and getting very impressive figures. On a Linux machine, I'm running fio tests and have the below config to test random read speeds -

[global]
name=fio-rand-read
filename=fio-rand-read
rw=randread
bs=4K
direct=0
numjobs=1
time_based
runtime=900


[file1]
size=10G
ioengine=libaio
iodepth=16

From this, I'm getting 35k IOPS, but the Ceph charts are nearly idling. So I'd imagine this is just hitting some kind of cache? If I set direct to = 1, the results are even more impressive. Is there any way that I can test from a VM what throughput I'm actually getting?

Thanks,
Chris.

 

[spirit]

I've been testing various disk caches with Ceph RBD. I'm using the writethrough cache mode and getting very impressive figures. On a Linux machine, I'm running fio tests and have the below config to test random read speeds -

[global]
name=fio-rand-read
filename=fio-rand-read
rw=randread
bs=4K
direct=0
numjobs=1
time_based
runtime=900


[file1]
size=10G
ioengine=libaio
iodepth=16

From this, I'm getting 35k IOPS, but the Ceph charts are nearly idling. So I'd imagine this is just hitting some kind of cache? If I set direct to = 1, the results are even more impressive. Is there any way that I can test from a VM what throughput I'm actually getting?

Thanks,
Chris.

direct=1 should already by used, or without it, you'll read the datas in the buffer cache of the os.

it's shouldn't be faster with direct=1.

about writethrough, specific to ceph lirbd, writeback|writethrough enable rbd_cache=true.
For write, it's really helping . I'm not sure about reads.

(I'm using writeback in prod)

 

[chrispage1]

Thanks @spirit

I'd be inclined to say the same regards direct=0 / direct=1but this is what I'm getting below...

root@testing:# cat fio-rand-read.fio
; fio-rand-read.job for fiotest

[global]
name=fio-rand-read
filename=fio-rand-read
rw=randread
bs=4K
direct=1
numjobs=1
time_based
runtime=900

[file1]
size=10G
ioengine=libaio
iodepth=16
root@testing:# fio fio-rand-read.fio
file1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.16
Starting 1 process
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [r(1)][2.7%][r=405MiB/s][r=104k IOPS][eta 14m:37s]

As much as I'd love our setup to run at 104k IOPS, there's no way that is correct!

Thanks,
Chris.

 

[spirit]

Thanks @spirit

I'd be inclined to say the same regards direct=0 / direct=1but this is what I'm getting below...

root@testing:# cat fio-rand-read.fio
; fio-rand-read.job for fiotest

[global]
name=fio-rand-read
filename=fio-rand-read
rw=randread
bs=4K
direct=1
numjobs=1
time_based
runtime=900

[file1]
size=10G
ioengine=libaio
iodepth=16
root@testing:# fio fio-rand-read.fio
file1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.16
Starting 1 process
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [r(1)][2.7%][r=405MiB/s][r=104k IOPS][eta 14m:37s]

As much as I'd love our setup to run at 104k IOPS, there's no way that is correct!

Thanks,
Chris.

how many osd do you have ? ssd ? nvme ?
do you have tried with a bigger size ? (try with 100-200GB).
also, try to bench direct a raw disk, on a second disk in the vm, with filename=/dev/sdb for example.


techically, 100k iops is also what I'm able to reach with ceph currently with 1 virtio-scsi disk. (limited by cpu on qemu side),
both with cache=none or cache=writeback. (on a full nvme cluster)