Network suggestion for 3 node cluster with 4 NICs

[cdel]

Hi there,

I want to build a 3 node cluster with ceph installed locally on each node. Unfortunately my Hardware is limited to 2x1G and 2x10G Interfaces. In my test environment I used the 10G interface in active/passive bond configuration, 1x1G for cluster network and vm2vm communication and 1x1G for uplink.
The Problem: no redundancy for cluster network and vm2vm communication. (uplink is solved with one virtual gateway routers on each node running vrrp). If the cluster network interface fails, the node looses quorum and VMs running on that node can't reach the others.
There is redundancy in the 2x10G interfaces, but everyone says "don't run corosync with storage traffic". Bonding the 2x1G and sharing uplink with cluster traffic is also not a good idea, I think.
An other option I was thinking of:
Using the 10G bond for storage and vm2vm, keep corosync on a single 1G and configure a 2nd ring for corosync on the 10G bond. This will keep a dedicated 1G link for corosync, but share the redundant link only if the other one fails.
Any other suggestions?

Best regards,
Christoph

 

[LnxBil]

Any other suggestions?

What about the obvious: Add another dual 10 GBE adapter to each node and run it like it should be.

I don't see that many problems with corosync and storage on 10 GBE with respect to running one of them "only" in a non-redundant setup. Yes, it's not optimal, but it'll work and you will not have a SPOF.

 

[cdel]

Hi LnxBil,

thanks for your reply. Unfortunately adding another dual 10 GBE adapter is not possible due to Hardware and Provider restrictions but we managed to get another pair of 1G interfaces into the Systems which should be fine as well.

Best regards,
Christoph