Design - Network for CEPH

[alfred.johansen]

Hi,

I'm new to the Proxmox world and I was wondering what would be the optimal way to build the Cluster network for CEPH for my type of setup.

Constraints/considerations
I do have some constraints and considerations noted below,

4 identical Hosts
128 total cores of AMD EPYC
NVME-disks
6x10 GB/s interfaces

2x UBNT USW-EnterpriseXG-24
Do keep in mind that these switches is not stacked.

The design must be redundant.
I plan on using SDN for the VMs.

Planned design
I'm planning on using 4 interfaces per host for the CEPH-network.
Connect these to the switches as below,


This way i would have redundancy if one switch fails and be able to use LACP(2 for each host).

I plan to use 2 interfaces per host for the SDN/external communication in LACP.

Questions

• Will 20 Gb/s be enough throughput with NVME-disks?
• Is the above design recommended/supported?
• Have i missed something? any improvements?

Greatful for your feedback on this!

Best Regards,
Alfred

 

[jdancer]

May want to look at various Ceph cluster benchmark papers online like this one https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2020-09-hyper-converged-with-nvme.76516/


Will give you an idea on design.

 

[sg90]

CEPH can only do one network per front and back, so you won’t be able to do redundancy unless you can do LACP across both physical switches.