What is "Normal" Ceph performance?

[maomaocake]

As the title suggests, I wanna find out what are the performance of other people's clusters.

I'll start:
Node Count: 3
Networking: 1 gbe shared with proxmox
Disks:

1. 3 x 3TB 7200RPM
2. 2 x 1TB 7200RPM
3. 1 x 1TB 5400RPM

using this Benchmarking Tutorial here's my result

 

[gurubert]

The performance is expected. Every write has to go through every node and you only have one 5400RPM HDD in the third node. This disk is hit with every write operation.

You should balance the disks (and the capacity) evenly across the nodes. This is important especially in such a small cluster. Move one 1TB and one 3TB disk to each node.

 

[maomaocake]

The performance is expected. Every write has to go through every node and you only have one 5400RPM HDD in the third node. This disk is hit with every write operation.

yes im aware of the performance i have a new 1tb 7200rpm hdd coming in the mail and moving my wal and db to a separate ssd, I just want to see if other people gets around as shitty performance as I do.

 

[gurubert]

I just want to see if other people gets around as shitty performance as I do.

I have seen worse performance on an all HDD cluster with very small S3 objects.
There was just too much action in RocksDB (and WAL) for the HDD to cope.