[SOLVED] Ceph configuration - best practices

[VoIP-Ninja]

Ceph speaks IP, which IPoIB provides. Though it doesn't seem to be such a common setup.

(and yeah, sadly somewhere on this forum I wrote incorrectly Ethernet)

this is what was in my mind, translated to IP not ethernet

 

[itret]

So I ended up with 40Gb network set to Ethernet. I didn't spend a lot on this setup so I may uprgade it. 100Gb is tempting

 

[eliyahuadam]

So I ended up with 40Gb network set to Ethernet. I didn't spend a lot on this setup so I may uprgade it. 100Gb is tempting

Hello @itret

I'm facing also poor write performance (with NVME OSD's - PM1733).

Ceph - poor write speed - NVME | Proxmox Support Forum

Do you mange the iprove the write speed with the Mellnox network card?

 

[itret]

Hello @itret

I'm facing also poor write performance (with NVME OSD's - PM1733).

Ceph - poor write speed - NVME | Proxmox Support Forum

Do you mange the iprove the write speed with the Mellnox network card?

Yes, network speed is crucial in this case. Also please bear in mind 40GBit Mellanox cards are set to InfiniBand by default, make sure you change configuration to Ethernet.

 

[eliyahuadam]

Hi @itret

Thanks for the information.
Which network card you use before?

Yes, network speed is crucial in this case. Also please bear in mind 40GBit Mellanox cards are set to InfiniBand by default, make sure you change configuration to Ethernet.

 

[eliyahuadam]

I've replaced the network card with Mellanox-connectx-6 25 gig

Unfortunately, the write speed still very poor.

Someone have any idea what can be the reason ?

Thanks.

 

[itret]

I've replaced the network card with Mellanox-connectx-6 25 gig

Unfortunately, the write speed still very poor.

Someone have any idea what can be the reason ?

Thanks.

Are you sure Ceph's config is set to use these cards?

 

[eliyahuadam]

Are you sure Ceph's config is set to use these cards?

Hi

The CPU c-state issue, i added the following lines to the GRUB:

intel_idle.max_cstate=0 processor.max_cstate=0

The servers configured with "high performance" profile (BIOS) but the c-state settings not reflected to the OS.

After rebooting the servers, the ping latency change to 0.02 instead of 0.2-3.
The write speed IOPS x3.5 better.

Thanks.