Interface Bonding - iSCSI Storage, corosync, ceph and VM Network - Best Practice?

[billyjp]

Hello together!

Maybe you can give me some tips?

I am planning a PVE cluster with 4 HPE servers following network cards per node:

4x onboard NICs (total 4x 1Gb)
4x Intel X520-DA2 PCIe (total 8x 10Gb)

Network:
2x switch with 28x 10Gb and 4x 25Gb (LACP 802.3ad is supported but unfortunately no MLACP).

Storage:
iSCSI SAN 2x 25Gb Nic. 1x 25Gb per switch connected (balance-alb)

Now I have the question how to configure the networks best:

Would like to use a secure and performant variant.

My idea would be:

Coronosync: 2x 10Gb with balance-alb (1x Switch1 1x Switch2)
iSCSI Storage: 2x 10Gb with balance-alb (1x Switch1 1x Switch2)
ceph Network: 4x 10Gb with balance-alb (2x Switch1 2x Switch2)
Management Network: 4x 1Gb with balance-alb (2x Switch1 2x Switch2)

What is your opinion?

Or for the Networks with 4x Ports should i use 2Ports LACP and bond then the 2x LACP Bonds together with balancing-alb?

 

[aaron]

Coronosync: 2x 10Gb with balance-alb (1x Switch1 1x Switch2)

Give it the interfaces raw with two different subnets. Corosync can use up to 8 links and will switch between them if needed to keep the cluster communication up and running. Then you can also configure the other networks as additional links as a failsafe.

ceph Network: 4x 10Gb with balance-alb (2x Switch1 2x Switch2)

In order to get more overall bandwidth? Just for your information (you can of course choose yourself), Ceph has the mandatory public network, but can also have the optional cluster network which will be used for the replication traffic between the OSDs. This way you could also split the load over two separate networks.

Management Network: 4x 1Gb with balance-alb (2x Switch1 2x Switch2)

Should this also be used for the guest traffic to the outside? Depending on your security concerns, you might want to separate mgmt and guest traffic completely. Though VLANs should also work.

(LACP 802.3ad is supported but unfortunately no MLACP).

Are the switches stacked? How would the network behave, if in a bond with 2 interfaces you have the situation that node A lost the connection to switch X, and node B lost the connection to switch Y? (Each node only connects to one switch, but not the same switch). Will they still be able to communicate?