Ceph low performance (especially 4k)

[sander93]

also, on ceph nodes, if they are not shared with vm, you can try to disable spectre/meldown fixes (don't known which kernel you use)

/etc/default/grub

GRUB_CMDLINE_LINUX=" pti=off spectre_v2=off l1tf=off spec_store_bypass_disable=off"

also, you can try to put your cpu frequency to always max + elevator noop

GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable quiet intel_idle.max_cstate=0 processor.max_cstate=1 elevator=noop"

I am running on CentOS 7: "3.10.0-862.11.6.el7.x86_64"

Can i add this option safe without problems?

 

[janos]

It's not about bandwith, but generally, 10GB switch have bigger asic with lowest latency.

(try to de a "ping -f" to compare between you promox<->osd, and proxmox<->mon)

Not the 10GBit matter, the media matter.

10 or 1 GBit with SFP over optical cable have much lesser latency than copper based link even if it that a 10gbit link. (not because of the media, the difference is beetwen how SFP and normal RJ45 interface working, SFP DAC cable also better than regular rj45 copper).

 

[sander93]

also, on ceph nodes, if they are not shared with vm, you can try to disable spectre/meldown fixes (don't known which kernel you use)

/etc/default/grub

GRUB_CMDLINE_LINUX=" pti=off spectre_v2=off l1tf=off spec_store_bypass_disable=off"

also, you can try to put your cpu frequency to always max + elevator noop

GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable quiet intel_idle.max_cstate=0 processor.max_cstate=1 elevator=noop"

This helps 4k a little bit, i now get 5,2 MB/s 4k Read and 2,6 MB/s 4k Writ!

But i think with this hardware this could me more right?

 

[Ibrahim Yashau]

Hi, just wondering.. did you ever find a fix for this?

 

[sander93]

Hi,

No not really, performance increased a little bit after a new switch bit still almost the same..

 

[p.jahn]

Hello,

I'm struggling with the same issue.

After some tuning I managed to increase 4k write IOPS from 539 (2158.2KB/s) to 1288 (5154.3KB/s) but I'm not sure if that's fine for a 10GE setup.
Maybe the latency is too high and 40GE would increase IOPS but it feels like there is a bottleneck somewhere in my setup.
Like Alwin said a SM863 would get about 17k IOPS while my SM863a gets only about 8k.

 

[Alwin]

@p.jahn, see our ceph benchmark paper for comparison.
https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/

 

[p.jahn]

Thanks @Alwin.
I have also found some rados bench results on the ceph mailinglist.
These were close to mine, so I think my performance is ok.
I have learned that it's all about latency and to increase the 4K IOPS I would need faster CPUs (higher clock) and network with lower latency like 40/100GbE.