ceph 5 node setup planning

[yaboc]

hi

i plan on building ceph setup of 5 nodes with proxmox corosync in mesh over 1Gb QUAD NIC, ceph pub/priv in mesh over 25Gb QUAD NIC and VM access network (also used as 2nd ring for proxmox coro) over 25Gb NIC connected to redundant 25Gb switches. We don't see ourselves expanding past 5 nodes anytime soon and opted for mesh setup.

Is it reasonable to have ceph pub and priv on one network (mesh)? I'm hoping that having direct connections over 25Gb shouldn't cause any issues during replication/re balancing.

It'll be HCI seup with 3x2TB OSD per node and 256GB RAM with decent dual CPU. I plan on testing this in virtual lab first but i was wondering if anyone had experience with mesh setup. The load isn't that crazy either, around 20VM 10-30GB 300GB-700GB storage. We will add SSD OSD (enterprise grade) to expand and stay within safe limits.

Thanks

 

[t.lamprecht]

See my reply to the other post:
https://forum.proxmox.com/threads/g...works-for-ceph-and-proxmox.58439/#post-312620

The OSDs being SSDs, or? around 500 - 600 MBps bandwidth? If so the network should hold up just fine, IMO.

 

[yaboc]

thanks for your reply t.lamprecht. i didnt wanna hijack the other thread and figured ill post this one with more HW info which you inquired about in your reply.

The OSD will most likely be Micron 5300 PRO 1.92 TB or open to suggestions, but my research so far narrowed it down to this model. I think they can push 500-600MBps with significant iops but ceph overhead will definitely take some toll on the transfer rates. it's all up for testing and tuning before going into prod.

so do you think keeping ceph pub/priv on one 25Gb mesh network should be sufficient ? i could potentially add 10gb switch somewhere down the line and use it for dedicated ceph private network.

 

[t.lamprecht]

so do you think keeping ceph pub/priv on one 25Gb mesh network should be sufficient ? i could potentially add 10gb switch somewhere down the line and use it for dedicated ceph private network.

Yes I think that should work out. If you really run into bandwidth, or more likely latency, issues I'd rather put the public network on 10G, the private on the faster one - the redundancy write and re-balancing happens there after all.
But as said, IMO the bandwidth you have is enough for 3 such OSDs per node, even if you get them up to 5 per node it should be still enough.
But yes, as real system definitively have often their own will, some good testing + tuning would be good to tell this for sure