Sep 27, 2019
With ceph - if you have a 5 node system.
With some nodes having dual NVME for CEPH, Samsung PRO

Why do they preform so badly compared to Intel Enterprise SSD or any other SSD?

The NVME drives show a latency of 1000+ compared to the SSD with a latency of 2+
Even on the same node.

On the other nodes with a Seagate Barracuda 120 SSD the latency is 2+
Even the Seagate FireCuda NVME has a latency of 10ms

are motherboard NVME drives not meant for CEPH?

please assume all Nodes are i9 CPU/motherboards / DDR5 Ram etc....


Consumer grade SSD's (including NVMes) performance usually drops in a considerable way once they fill their cache. Once you are in this state latency will also go up.

Ceph does more IO operations than other filesystems like ext4 and it is also more sensitive to latency, so you are more likely to notice the drop in performance.

If you want some numbers you can take a look at one of our benchmarks [1].

[1] https://www.proxmox.com/images/download/pve/docs/Proxmox-VE_Ceph-Benchmark-202009-rev2.pdf
brilliant response, exactly what I needed.
many thanks


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!