Proxmox and CEPH - Any options?

[dpearceFL]

Are there are configuration options or performance tuning available for CEPH?

 

[waltar]

Normally more ceph server and more ceph osd's but anyway throughput in a distributed filesystem going up with multiple client requests while single client requests were relatively low.

 

[alexskysilk]

Are there are configuration options or performance tuning available for CEPH?

yes. lots.

and to forestall your next question, they're in the documentation but more to the point they're not relevant until you define what it is you're trying to accomplish.

 

[aaron]

Without much infos to go on. To get good performance on Ceph, you need a few things:

• fast datacenter SSDs with PLP
• low latency network with enough bandwidth. Fast SSDs can make the network quickly a bottleneck, see Ceph 2023 Benchmark Paper
• configure BIOS to low latency / high performance settings. Check your hardware vendor for any guides. Disable any power savings.
• enough CPU cores for each Ceph service to avoid CPU overprovisioning
• Target-Ratio per pool configured for the autoscaler to be able to estimate the optimal number of PGs per pool, so that the PGs/OSD is in the ballpark of around 100

With these settings/HW the performance should be fine.

 

[alexskysilk]

• fast datacenter SSDs with PLP

Not to be pedantic, but ceph isnt dependent on osd level plp and would work fine without it

 

[aaron]

No, but OSDs will issue a lot of sync writes, which these SSDs will handle the best regarding performance, as they can ACK the write operation once the data is in the local cache/RAM of the SSD.
Which is why you will see in the Ceph docs a section on how to enable the cache on HDDs to improve performance.

 

[alexskysilk]

Which is why you will see in the Ceph docs a section on how to enable the cache on HDDs to improve performance.

Absolutely FANTASTIC tip. I went to look through some of my osds and found a BUNCH with WC turned off

While not PLP related, pro tip: TURN ON YOUR OSD WRITE CACHE.