question about bond count on Ceph nodes

[kajatonas]

Hello,
In our setup every server has 2 NICs with 2 interfaces each. For simplicity and for saving network devices resources we want to bond all 4 interfaces to one bond interface, not the two as it written in some PDFs.
So there would be one bond with 4x10GBs. Is there any limitations to this setup instead 2 bond (2x10GB each) ?
In some manuals I've seen that it's specifying the configuration where one bond has public vlans and other one replication based vlans. So, is there any pain if we would add all vlans to one bond instead of two ?

Thanks !

 

[Klaus Steinberger]

It is recommended to separate CEPH backend traffic from Frontend, because of latencies.

also a 4x Bond will not help you in terms of speed as long as you do not have at least 5 nodes.
in the usual bonding method (LACP) the usage of the bond links is done on IP/MAC, so for example Node A talks with Node B over LINK A but with Node C over Link B (and so on)

So between two specific nodes you will have not more than 10 Gbit/s speed (but of course redundancy)

So my advise is to setup 2 bonds with 2 interfaces each.

 

[czechsys]

Two points:
1a] in one-bond mode if bond fail, you lost all conectivity
1b] in two-bond mode if one bond fail, you still have something left (usabiliy depends...)
2a] in one-bond mode bond sides select, what iface will be used to send datastream
2b] in two-bond mode admin decide, what ifaces will be used by specific datastreams

 

[Klaus Steinberger]

Two points:
1a] in one-bond mode if bond fail, you lost all conectivity
1b] in two-bond mode if one bond fail, you still have something left (usabiliy depends...)
2a] in one-bond mode bond sides select, what iface will be used to send datastream
2b] in two-bond mode admin decide, what ifaces will be used by specific datastreams

That depends heaviily on the modes supported by both operating system and the switch side!

not all static bond modes work with every switch and can lead to very annoying errors. With many switch brands only active/passive bonding works reliable.

LACP is more dynamic and detects failures of ports better, but uses under linux a Hash over IP and MAC Adress to select which port is used for transmission. So don't expect more than the speed of one wire for a connection between two specific nodes.
Of course the switch has to support LACP and has to be configured correctly.