Slow Sequential Write Performance on Ceph RBD

[nicolasyang]

I am operating a 3-node cluster. I ran a fio benchmark in a VM with following configurations:

size: 1g
iodepth: 64
ioengine: linux aio
block size: 1M

And the result is 142 MB/s.

I ran a fio benchmark directly with librbd, outside of VMs, and got a result of 266 MB/s. And a rados bench get a result of 508 MB/s.

The write performance inside VMs seems too slow comparing to librbd or rados perforamce. I'm not sure why

 

[Alwin]

See our Ceph benchmark paper and the corresponding forum thread for reference.
https://www.proxmox.com/de/downloads/item/proxmox-ve-ceph-benchmark
https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/

 

[donalddk]

Did you manage to get better performance ?

 

[nicolasyang]

Did you manage to get better performance ?

Yeah. Just use krbd instead of librbd.