Poor randwrite fio performance result

[tincboy]

I've a SSD based Ceph cluster with 3 nodes, the read IOPS is about 250K with 96 parallel fio jobs ( running from 3 different nodes ), the reasults are fine.
But the Write performance is not more than 2k IOPS when more than 1 parallel jobs is running. ( with only one job I can reach 15K write IOPS )
My concentration is now on improving write IOPS.
is there anything I should consider when testing with fio the randwrite scenario ?

[global]
#logging
ioengine=rbd
clientname=admin
pool=tests
rbdname=fio_test
invalidate=0    # mandatory
rw=randwrite
bs=4k
time_based
runtime=60
numjobs=1

[jobinfo]
iodepth=100

 

[spirit]

The problem is that with multiple jobs with 1rbd, you have the exclusive_lock feature on rbd volume(only write from 1 client/job at same moment).

you can try to test with multiple rbd volumes, on try to desactivate exclusive_lock feature (for testing only) on the rbd volume.

 

[tincboy]

The problem is that with multiple jobs with 1rbd, you have the exclusive_lock feature on rbd volume(only write from 1 client/job at same moment).

you can try to test with multiple rbd volumes, on try to desactivate exclusive_lock feature (for testing only) on the rbd volume.

Thanks, you are right indeed.
I've figured it out by "--image-shared" option when creating fio image, like code below:

rbd -p rbd create --size 2048 fio_test --image-shared