Expected Ceph performance?

[kez]

Hi,

What is the 'normal/expected' VM disk performance on Ceph?

In this instance, it's;

- 4 x nodes each with 1 x NVMe (3000MB/s ish)
- Dedicated Ceph network
- 20G bonded links between nodes/switch (iperf 17.5Gbit/s)
- MTU jumbo

Here is an example rdb bench test (lowest 289 MiB/s):

root@box1node1:~# rbd bench --io-type write --io-size 4096 --io-threads 16 --io-total 1G NVMe1/vm-107-disk-0
bench  type write io_size 4096 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1    108720    108844   425 MiB/s
    2    169648   84873.9   332 MiB/s
    3    221744   73944.2   289 MiB/s
elapsed: 3   ops: 262144   ops/sec: 68337.5   bytes/sec: 267 MiB/s

Best I can get out of a VM: (776MiB/s ish) after playing with disk controller (VirtIO SCISI single) and settings: (IO Thread = yes, Aysinc IO = threads).

root@test:~# fio --name=randrw --ioengine=libaio --rw=randrw --bs=1M --size=2G --numjobs=8 --iodepth=32 --runtime=60 --group_reporting
randrw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.36
Starting 8 processes
randrw: Laying out IO file (1 file / 2048MiB)
randrw: Laying out IO file (1 file / 2048MiB)
randrw: Laying out IO file (1 file / 2048MiB)
randrw: Laying out IO file (1 file / 2048MiB)
randrw: Laying out IO file (1 file / 2048MiB)
randrw: Laying out IO file (1 file / 2048MiB)
randrw: Laying out IO file (1 file / 2048MiB)
randrw: Laying out IO file (1 file / 2048MiB)
Jobs: 8 (f=8): [m(8)][91.7%][r=739MiB/s,w=745MiB/s][r=739,w=745 IOPS][eta 00m:01s]
randrw: (groupid=0, jobs=8): err= 0: pid=997: Sun Jul 20 18:50:44 2025
  read: IOPS=745, BW=746MiB/s (782MB/s)(8029MiB/10768msec)
    slat (usec): min=2317, max=31799, avg=8730.15, stdev=4620.60
    clat (usec): min=7, max=253162, avg=159347.48, stdev=28721.48
     lat (msec): min=3, max=257, avg=168.08, stdev=29.21
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[  115], 10.00th=[  126], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  155], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  182], 90.00th=[  194], 95.00th=[  203],
     | 99.00th=[  220], 99.50th=[  226], 99.90th=[  241], 99.95th=[  243],
     | 99.99th=[  253]
   bw (  KiB/s): min=552609, max=936334, per=99.39%, avg=758857.50, stdev=12304.77, samples=167
   iops        : min=  538, max=  912, avg=739.09, stdev=12.00, samples=167
  write: IOPS=775, BW=776MiB/s (814MB/s)(8355MiB/10768msec); 0 zone resets
    slat (usec): min=572, max=15688, avg=1761.55, stdev=1877.55
    clat (usec): min=4, max=253215, avg=160838.88, stdev=28904.62
     lat (usec): min=704, max=259034, avg=162600.43, stdev=29025.51
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[  116], 10.00th=[  127], 20.00th=[  140],
     | 30.00th=[  148], 40.00th=[  157], 50.00th=[  163], 60.00th=[  169],
     | 70.00th=[  176], 80.00th=[  184], 90.00th=[  197], 95.00th=[  205],
     | 99.00th=[  222], 99.50th=[  228], 99.90th=[  241], 99.95th=[  245],
     | 99.99th=[  253]
   bw (  KiB/s): min=592879, max=1012723, per=99.66%, avg=791840.27, stdev=13478.95, samples=167
   iops        : min=  578, max=  988, avg=771.21, stdev=13.18, samples=167
  lat (usec)   : 10=0.04%, 50=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=0.09%, 20=0.10%, 50=0.39%
  lat (msec)   : 100=1.14%, 250=98.18%, 500=0.02%
  cpu          : usr=1.09%, sys=15.71%, ctx=12363, majf=0, minf=113
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=8029,8355,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=746MiB/s (782MB/s), 746MiB/s-746MiB/s (782MB/s-782MB/s), io=8029MiB (8419MB), run=10768-10768msec
  WRITE: bw=776MiB/s (814MB/s), 776MiB/s-776MiB/s (814MB/s-814MB/s), io=8355MiB (8761MB), run=10768-10768msec

Disk stats (read/write):
  sda: ios=7997/8216, sectors=16375808/16720064, merge=0/17, ticks=49057/135099, in_queue=184156, util=52.94%

Thanks for any advice on this.
C