3 node test setup ceph issues

[magicker]

Hi all

I have 3 nodes each with 4x256gb ssds.
I have set all up as a cluster and installed ceph. As this is a test setup everything shares the same 1gb internet connection over the internet.
When the clusters we setup with hardware raid 10 this worked great and migrating VMs between clusters was very good.

Next step is to test the ceph setup. So raid was disabled. Proxmox installed on 1 of the drives. The other 3 drives on each server was added as OSD to ceph. So 9 drives in cluster.

However, the benchmarks were terrible..

rados -p test bench 10 write --no-cleanup
Bandwidth (MB/sec): 32.9994

rados -p test bench 10 seq
Bandwidth (MB/sec): 83.1497

Installing a windows vm took about 2 hours. (it is still "getting ready as I type)
Just for laughs I created an identical using the local storage.. this installed in around 5 mins.

i/o delay hovers around 15%

Is this about right for a 1gb link or have I done something stupid? am I just missing the 10gb link between the nodes and everything will burst back to life?

 

[ness1602]

With 1gbps link you are limited to 1gbps/3 at most, so this is around 112MB/s / 3, eq around 35MB/s write, so this is okay

 

[Cookiefamily]

I have a cluster for testing with three nodes and 1G links as well... And yes, performance sucks In production we use 2x10G and the performance is decent. Can't recommend CEPH with 1G links even using LACP with multiple 1G links is not fun as LACP is session based. 1G also has a higher latencies than 10G/25G/40G/100G, see https://www.blog.adva.com/en/lets-take-bit-time-talk-about-low-latency

 

[sidereus]

In production we use 2x10G and the performance is decent.

Could You share your ceph.conf? I'm interesting also try 2x10G

 

[magicker]

I have a cluster for testing with three nodes and 1G links as well... And yes, performance sucks In production we use 2x10G and the performance is decent. Can't recommend CEPH with 1G links even using LACP with multiple 1G links is not fun as LACP is session based. 1G also has a higher latencies than 10G/25G/40G/100G, see https://www.blog.adva.com/en/lets-take-bit-time-talk-about-low-latency

Is that 2x10gb into a 10gb switch?

 

[Cookiefamily]

Is that 2x10gb into a 10gb switch?

yes, 6 proxmox/ceph nodes each with 2x10G connected to redundant switches. As 25G is getting pretty cheap by now, new servers already come with 25g interfaces, we just haven't upgraded the switch yet.

 

[Cookiefamily]

Could You share your ceph.conf? I'm interesting also try 2x10G

What do you want to know from that ceph.conf? networks and keyrings are irrelevant and other than default values there isn't really a ton in there.

 

[sidereus]

What do you want to know from that ceph.conf? networks and keyrings are irrelevant and other than default values there isn't really a ton in there.

mon_host - how to specify 2 addresses for each node?

 

[Cookiefamily]

mon_host - how to specify 2 addresses for each node?

the nodes just have one mon addr in our case