ceph bluestore much slower than glusterfs

[svirus]

Hello,

I just want to create brand new proxmox cluster.
On some older cluster I used glusterfs, now I have some time and I try to compare glusterfs vs new ceph (PVE 5.2).
on my lab I have 3 VM (in nested env) with ssd storage.
iperf show between 6 to 11 gbps, latency is about 0.1ms

I make one storage with ceph, and one with glusterfs, then I run one VM on ceph, and on Gluster (latest debian on both, and same configuration).

VM on Gluster can make up to 4k iops and transfer up to 10MB/s
VM on Ceph can make up to 100 iops and transfer up to 1MB/s

so it's slowest more than 10 times :/

i know is nested env. but this diffrents is too big to make it on production.

Thanks in advanced, and sorry for my english.

 

[sseidel]

Sorry if I can't help directly, but yes, Ceph is slower than expected. You can speed up Ceph by using krbd.

Your numbers are very slow for SSD. We have a production system with SSD and 10GbE, read/write is around 150MB/s, IOPS are around 1000-2000.

Test system is on 1GbE public network, 2GbE cluster network and HDD, we get around 300-400 IOPS.

Maybe you have a wrong cache setting for your test nested hosts?

 

[wolfgang]

Hi,

VM on Gluster can make up to 4k iops and transfer up to 10MB/s
VM on Ceph can make up to 100 iops and transfer up to 1MB/s

so it's slowest more than 10 times :/

Sorry to say but your comparison is not vailed and say nothing.
This kind of test is the same as HW vendor X say my disk is 100% faster in case x than bla.
The problem is case X is one case from more than one.
GlusterFS is based on a FS and not on blockdev, what makes a huge difference on nested environments.