Proxmox CEPH - SSD Journal and HDD (5400rpm) OSD

[KHosting]

I am building a Proxmox CEPH cluster (3nodes) and does not have a budget for SSD because it will be about 8T per node. So I thought of using high speed Enterprise SSDs for Journaling and then 4 x Seagate Spinpoint M9t 2Tb 2.5" ST2000LM003 for OSD per node

This disk is only a 5400RPM and that got me concerned.

Does anyone has any experience using this drive or possible recommend a more possible alternative?

Will the Enterprise SSD Journal and HDD OSD Proxmox CEPH really produce a good speed network?

Thank you.

 

[robhost]

With this slow HDDs Ceph will have a really bad read performance (writes will also be slow, escpecially when flushing from the journal to the HDDs). It depends on your workload, but with many VMs this can become a problem.

 

[e100]

I can give you some real-world examples using mechanical disks on some older hardware.

I've got three CEPH nodes with 6 disks each, all 7200 RPM SATA disks.
12 are 4TB and the rest are mix of 1/2/3TB disks.
We did move journals to SSD but for a single VM it did not make too much of a difference in write performance.

Write performance inside a single VM is 30-40MB/sec Read is 20-30MB/sec
If the data being accessed is in cache RAM on the CEPH server read access is orders of magnitude faster.
But if the OSD needs to read from the disk the latency is so high that performance is as described.

Ceph benchmarks are much better because it can use multiple threads where KVM is limited to a single thread.
If KVM had multi-threaded IO I suspect it too would perform much better.

I also think my older CPUs on the Proxmox and CEPH servers is partly responsible for my lower performance.
With a single thread latency can kill performance, faster CPU reduces latency.

We have Intel E5420 on Proxmox nodes and AMD Phenom II 1090T on CEPH nodes.

Before picking an SSD for journals read this:
https://www.sebastien-han.fr/blog/2...-if-your-ssd-is-suitable-as-a-journal-device/

 

[KHosting]

Thank you for your feedback @robhost and @e100.

Write performance inside a single VM is 30-40MB/sec Read is 20-30MB/sec

This seem very low. So I am going to be using a 2 x E5-2660 per node. Would that really help improve performance?