From my last bench, with fio on rbd,Do you think this is as expected (maybe 4k write could go faster?)? With 2000 more bucks we could do a 40 GBit mesh network. Do you think it's worth trying this?
I'm able to reach around 150000 iops 4k randwrite, limitation is the cpu of the ceph nodes. (3 nodes with 24cores 3ghz, 100% cpu).
For read, I'm around 700000 iops 4k randread, cpu is limiting too. (3 ceph nodes 100% cpu, and 2 clients nodes (same cpu config) cpu 100%).