Configurating cluster networks with Ceph?

[Birdie1]

Hi everyone!
I recently read a lot of stuff about hyperconverged PVE/Ceph clusters and now struggle to see the wood through all the trees. ;-)
I (plan to) have a three node Proxmox cluster with 4 NICs in each node (2x 1Gbe, 2x 10 Gbe) and would like to have some advice on which NIC should be dedicated to which service.essay writer
The first one is easy: one 1 Gbe for corosync, maybe with a fallback on one 10 Gbe NIC (which is connected to another switch)
Next, I would dedicate one 10 Gbe for VM traffic to the LAN
And last, I would dedicate the other two NICs for Ceph, one external and one cluster network.

Now, does this make sense? If so, would the 10 Gbe NIC become dedicated either for the external or for the cluster network? Would an active/backup bond between those two NICs maybe make more sense, given that they are connected to two different switches?

I appreciate any input and also further reading material.

 

[Pierre-Yves]

Ceph speed really depends on network bandwith :
https://www.mellanox.com/related-do...oying_Ceph_over_High_Performance_Networks.pdf

An interesting paper about Ceph (with Openstack but it doesnt matter in fact)
http://www.cs.utsa.edu/~atc/pub/C51.pdf

If you plan to agregate network cards, ou can only use same speed cards and full duplex mode, you cannot bond a 10Gbs and a Gigabit card in a LACP bond.

Regards

 

[czechsys]

My cheap suggestion:
1] LACP 802ad 2x 1Gbps - backup corosync vlan + pve vlan (=ceph monitors too) + vm vlans access
2] LACP 802ad 2x 10Gbps - primary corosync vlan + ceph cluster (osds)

My standard cheap suggestion:
- use 2x2 10Gbps port for 1]+2] and spare 1 Gbps for primary corosync

And don't use rj45 for 10Gbps, only DAC or fibre.