3 node test setup ceph issues

magicker

New Member
Apr 11, 2021
2
0
1
50
Hi all

I have 3 nodes each with 4x256gb ssds.
I have set all up as a cluster and installed ceph. As this is a test setup everything shares the same 1gb internet connection over the internet.
When the clusters we setup with hardware raid 10 this worked great and migrating VMs between clusters was very good.

Next step is to test the ceph setup. So raid was disabled. Proxmox installed on 1 of the drives. The other 3 drives on each server was added as OSD to ceph. So 9 drives in cluster.

However, the benchmarks were terrible..

rados -p test bench 10 write --no-cleanup
Bandwidth (MB/sec): 32.9994

rados -p test bench 10 seq
Bandwidth (MB/sec): 83.1497

Installing a windows vm took about 2 hours. (it is still "getting ready as I type)
Just for laughs I created an identical using the local storage.. this installed in around 5 mins.

i/o delay hovers around 15%

Is this about right for a 1gb link or have I done something stupid? am I just missing the 10gb link between the nodes and everything will burst back to life?
 
With 1gbps link you are limited to 1gbps/3 at most, so this is around 112MB/s / 3, eq around 35MB/s write, so this is okay :D
 
  • Like
Reactions: magicker

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!