Why my CEPH is so slow?

[itret]

Hi there.

So I went thru many, many posts on this forum but I cannot find some working solution for my CEPH.

Here is my configuration

Cluster and ceph operate on one 10G network, VMs use separate 1G networks
I have 3 nodes setup where each physical machine has 256GB of RAM (usage max 70%)
For OSDs I use NVMe 2TB drives (I have 5 in total but for tests I've added only 3 of them)

I moved one VM to CEPH storage to check read and write speeds and I'm bit shocked how slow it is. Read about 133MB/s and write about 30MB/s

I have the most recent version of ceph 18.2 and proxmox 8.1.4

What am I doing wrong? Please

 

[rj45]

holy shit.. ) what's iperf results on 10G?

 

[Maximiliano]

Hello, could you please share with us

- The bits relevant to the Ceph network in `/etc/network/interfaces`?
- The contents of /etc/pve/ceph.conf
- The model of the OSDs
- The output of the command `ceph osd df tree`
- The VM config at /etc/pve/qemu-server/<VMID>.conf` used for testing if you used a VM for testing

 

[itret]

TCP window size:  102 KByte (default)
------------------------------------------------------------
[  1] local 192.168.98.52 port 33414 connected with 192.168.98.53 port 5001 (icwnd/mss/irtt=14/1448/63)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0301 sec  10.2 GBytes  8.72 Gbits/sec

 

[ness1602]

You didn't say what model are NVME drives?

 

[itret]

You didn't say what model are NVME drives?

WD RED SN700

 

[ness1602]

Yeah,these are pretty terrible for ceph.

 

[Nuke Bloodaxe]

@ness1602 Looking at the spec sheets at https://www.westerndigital.com/en-au/products/internal-drives/wd-red-sn700-nvme-ssd?sku=WDS200T1R0C you'd think they were reasonable, what's the specific reason they fall off a cliff?

 

[ness1602]

DPWD is around 1.3, for enterprise it is 3-5 DPWD.
Don;t have PLP.
And probably terrible fsync performance which is important for ZFS and CEPH within Proxmox.

 

[alexskysilk]

For OSDs I use NVMe 2TB drives (I have 5 in total but for tests I've added only 3 of them)

you want equal capacity on all three nodes. any excess capacity would not be usable.

 

[itret]

@ness1602 Looking at the spec sheets at https://www.westerndigital.com/en-au/products/internal-drives/wd-red-sn700-nvme-ssd?sku=WDS200T1R0C you'd think they were reasonable, what's the specific reason they fall off a cliff?

Ok. Good point, I didn't take a closer look on these. Which one you'd suggest in this case?

 

[ness1602]

Enterprise SSDs, so Intel,Micron,Samsung,Kioxia(ex Toshiba), so the ones that you know are suitable for enterprise, and in short won't fry fast.

 

[Maximiliano]

Please note that for an enterprise SSD we understand a disk that has power-loss protection, there might be a few exceptions but it works well enough as a rule of thumb.

Many brands offer those. Unfortunately, there are disks without PLP branded as enterprise in the market.

 

[ness1602]

Yes,there are some NAS ssd's with PLP, if i'm not mistaken old seagate ones, but usually look for PLP drives.

 

[itret]

Please note that for an enterprise SSD we understand a disk that has power-loss protection, there might be a few exceptions but it works well enough as a rule of thumb.

Many brands offer those. Unfortunately, there are disks without PLP branded as enterprise in the market.

I just ordered SAMSUNG PM983 with PLP, it's 1.3 DPWD tho. I couldn't find any m.2 drive with 3-5.