Best Utilization for network ports

[Frosty81]

I am building out a 3 node cluster for home lab use. Each node has (2) 2.5 gb nics which currently plug into 1 gb ports on my switch, (2) SFP+ ports, and (2) 40 gb ports. I currently have the 40 gb configured in a mesh network. I would like to separate everything out to avoid network bottlenecks as much as possible.

My thoughts for which nic to use for which service (correct me if there are incorrect assumptions or better ways to do any of it)

Proxmox Management: 2.5G
Multi Vlan VM's: SFP+
Ceph Private : 40G
Ceph Public: 2.5G

Should the ceph public and private share the same connection, or is breaking them up good? Does public need more than 1 gb connection, or is that plenty?

 

[shanreich]

The public Ceph network is what the VMs will use to access their storage. The private network is mainly used for replication traffic. It might make sense to not split those networks after all, since the public network will bottleneck the private network. You should get better overall performance if you just use the 40G as both the public/private Ceph network.