Clarification Regarding RBD Clients

Tmanok

Renowned Member
Hi Everyone,

I'm about to deploy a sizable CEPH environment for proof of concept purposes and I want to consider benchmarks vs real world usage. Therefore, I have a few clarifying questions that I don't think my shiny PVE Advanced Certification answered:
  • What counts as an RBD client in PVE?
    • Is each resource/guest (VM/LXC) an RBD client?
    • Is each guest virtual disk (or mount point) an RBD client?
  • How are the number of clients configured?
    • Are their limits per PVE Node or Cluster?
    • Can guests be configured to use more clients (per guest or per virtual disk?)
  • What benchmarking guidance would you offer me?
Additionally, I've had some thoughts regarding future production CEPH clusters:
  • At what point do I need/want to have bare metal / separated monitor nodes? (as seen in Supermicro's setup for example)
  • Do monitor nodes really need as much RAM as some of these benchmark setups suggest? My experience tells me Supermicro's approach is perhaps overkill but NVMe is a different animal to me and we're moving in that direction...
  • Many of the setups suggest 5 nodes but only say 60 OSDs total are added. I am looking at 80-90 OSDs approximately across 4 nodes. If the memory is sufficient, does this seem acceptable or would it be better to go with more nodes?

I've read the following CEPH Benchmarking guides for reference:
But none of them (except UTSA's sadly) talk about using any sort of real world virtualization workloads or databases etc. Certainly nothing answered my questions specifically targeted at Proxmox Virtual Environment. Note that I will be throwing workloads onto this storage cluster but I need to gain a better understanding of how PVE (or KVM/QEMU and PVE oddities) behave and interact with CEPH.

Thanks for any answers, guidance, or help you can offer.

Tmanok
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!