Ceph very slow rebalancing ~300Kib

[Y0ngg4n]

Hi i have recreated a osd in my hyperconverged cluster. I have a 10Gbit link so it should be really fast to rebalance. But it seems like to rebalance only with some kilobytes:


I have already set

ceph tell 'osd.*' injectargs '--osd-recovery-max-active 4'
ceph tell 'osd.*' injectargs '--osd-max-backfills 16'

But does not seem to change something.

Maybe somebody can help me?

 

[jdancer]

I get 100's in write IOPS and read IOPS are 2x-3x of write IOPS using SAS 10K RPM HDDs. This is with a 5-node Dell 12th-gen 16-drive bay servers using 10GbE.

I am guessing you are using consumer SSDs? They bottleneck real quick once their internal cache is filled. You'll want enterprise SSD with PLP ( power-loss prevention.

 

[Y0ngg4n]

No the osds are 3x Crucial NVME M2 PCI 3 2TB drives. So they should not be slow also with a 10GbE connection.

 

[_gabriel]

No the osds are 3x Crucial NVME M2 PCI 3 2TB drives. So they should not be slow also with a 10GbE connection.

seems P1/P2/P3 consumer model...
What's model ?

 

[Y0ngg4n]

Crucial P3 2TB M.2 PCIe Gen3 NVMe Interne SSD, Bis zu 3500MB/s - CT2000P3SSD8​

 

[tom]

Crucial P3 2TB M.2 PCIe Gen3 NVMe Interne SSD, Bis zu 3500MB/s - CT2000P3SSD8​

This is a cheapest consumer SSD, not suited for Ceph. (You will see countless reports here in the community with similar issues with consumer hardware and Ceph)

 

[tom]

=> start reading on https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2020-09-hyper-converged-with-nvme.76516/

 

[Y0ngg4n]

Ok can you give me an example of a good nvme for ceph?
It seems like currently the rebalancing settled at 20-30MiB per second
What is a normal speed?

 

[aaron]

Anything that has Power-loss-protection (PLP) is a good start. See for example: https://geizhals.at/?cat=hdssd&xf=4643_Power-Loss+Protection

Since the SSD has capacitors to provide enough power to write everything in cache down to non-volatile memory, it can ACK sync writes much quicker. Ceph issues a lot of sync writes.

 

[Y0ngg4n]

thank you for your help

 

[dagutman]

I am having a similar issue.. oddly though initially it was recovering at 500-800 MB/s and now it's slowed down to 5% of that.. All of my volumes (even slow SATA drives) are backed by NVME, although they are not (yet) the enterprise grade (which I am installing next week).

For my bulk storage though, I am using 3.5 inch SATA drives; they are all enterprise quality drives, but don't have capacitors like an SSD can have. For these scenarios, would using at least an Enterprise SSD for the WAL enhance performance? Trying to get a handle on this before I reconfigure everything.