CEPH - NVME

[damonpillinger]

With ceph - if you have a 5 node system.
With some nodes having dual NVME for CEPH, Samsung PRO

Why do they preform so badly compared to Intel Enterprise SSD or any other SSD?

The NVME drives show a latency of 1000+ compared to the SSD with a latency of 2+
Even on the same node.

On the other nodes with a Seagate Barracuda 120 SSD the latency is 2+
Even the Seagate FireCuda NVME has a latency of 10ms


Question:
are motherboard NVME drives not meant for CEPH?


please assume all Nodes are i9 CPU/motherboards / DDR5 Ram etc....

thanks
damon

 

[Maximiliano]

Hello,

Consumer grade SSD's (including NVMes) performance usually drops in a considerable way once they fill their cache. Once you are in this state latency will also go up.

Ceph does more IO operations than other filesystems like ext4 and it is also more sensitive to latency, so you are more likely to notice the drop in performance.

If you want some numbers you can take a look at one of our benchmarks [1].

[1] https://www.proxmox.com/images/download/pve/docs/Proxmox-VE_Ceph-Benchmark-202009-rev2.pdf

 

[damon1]

brilliant response, exactly what I needed.
many thanks
Maximiliano

 

[spirit]

be carefull that samsung "pro" is a consumer grade drive without supercapacitor. ("pro" is really bullshit marketing)

 

[damonpillinger]

be carefull that samsung "pro" is a consumer grade drive without supercapacitor. ("pro" is really bullshit marketing)

100% agree.
The Intel Enterprise SSD wipe the floor with them.
Thanks