Clarification Regarding RBD Clients

[Tmanok]

Hi Everyone,

I'm about to deploy a sizable CEPH environment for proof of concept purposes and I want to consider benchmarks vs real world usage. Therefore, I have a few clarifying questions that I don't think my shiny PVE Advanced Certification answered:

• What counts as an RBD client in PVE?
  • Is each resource/guest (VM/LXC) an RBD client?
  • Is each guest virtual disk (or mount point) an RBD client?
• How are the number of clients configured?
  • Are their limits per PVE Node or Cluster?
  • Can guests be configured to use more clients (per guest or per virtual disk?)
• What benchmarking guidance would you offer me?

Additionally, I've had some thoughts regarding future production CEPH clusters:

• At what point do I need/want to have bare metal / separated monitor nodes? (as seen in Supermicro's setup for example)
• Do monitor nodes really need as much RAM as some of these benchmark setups suggest? My experience tells me Supermicro's approach is perhaps overkill but NVMe is a different animal to me and we're moving in that direction...
• Many of the setups suggest 5 nodes but only say 60 OSDs total are added. I am looking at 80-90 OSDs approximately across 4 nodes. If the memory is sufficient, does this seem acceptable or would it be better to go with more nodes?

I've read the following CEPH Benchmarking guides for reference:

• Both PVE CEPH Benchmark PDFs
• Supermicro's CEPH Benchmark PDF
• Dell's CEPH Benchmark PDF
• UTSA's Benchmark PDF (oof)
• CERN's Old 2015 Benchmark PDF

But none of them (except UTSA's sadly) talk about using any sort of real world virtualization workloads or databases etc. Certainly nothing answered my questions specifically targeted at Proxmox Virtual Environment. Note that I will be throwing workloads onto this storage cluster but I need to gain a better understanding of how PVE (or KVM/QEMU and PVE oddities) behave and interact with CEPH.

Thanks for any answers, guidance, or help you can offer.

Tmanok