Ceph network failover

[Valdri]

Hi there,
I have 3 nodes and I want to use thunderbolt for ceph network.
I setup mesh network using openfabric according to this guide
As thunderbolt networking is a bit finicky, I want to have a failback connection for ceph network.

In ip route openfabris is sets static route e.g

10.15.15.1 nhid 12 via 10.15.15.1 dev thunderbolt0 proto openfabric metric 20 onlink

Can I somehow add a fallback route for 10.15.15.1 to be routed through vmbr0?

 

[freebeerz]

Hi, I just set up a ceph ring mesh network on 3 nodes using the "simple routed setup" from the guide you posted (2 TB ports on each node, connected to the other 2 nodes), and I was wondering the same thing about using a spare ethernet interface connected to a dedicated switch as failover or even aggregation with the meshed thunderbolt interfaces...

LACP would require ethernet switch support so not suitable for thunderbolt and I don't think it would be possible to create an active/passive bond with each TB port and the same ethernet failover link...

Or maybe it would be possible with VLANs on top of the single ethernet (1 TB port + 1 dedicated ethernet VLAN for each bond) ?

Anyone tried that?

 

[ucholak]

hi,
you cant use bonding on tb.
i have implemented this functionality with frr (ospf+ospfv6) + tb4 + lan (backup for ceph ip's).

in frr i just defined vmbr0 (on standard lan port) as backup link with higher cost.
enoX is tb4 with lower cost.

# Backup links via primary gigabit link (vmbr0)
# Cost for 1G assumptions (100 gig reference / 1 gig = 100 cost)
!
interface vmbr0
ip ospf area 0.0.0.0
ip ospf network broadcast
ip ospf cost 100
ip ospf authentication message-digest ..
ip ospf message-digest-key ...

ipv6 ospf6 area 0.0.0.0
ipv6 ospf6 network broadcast
ipv6 ospf6 cost 100
ipv6 ospf6 authentication key-id ..


!
interface en01
ip ospf area 0.0.0.0
ip ospf network point-to-point
ip ospf cost 10
ip ospf authentication message-digest ..
ip ospf message-digest-key ...


ipv6 ospf6 area 0.0.0.0
ipv6 ospf6 network point-to-point
ipv6 ospf6 cost 10
ipv6 ospf6 authentication key-id ..


!
interface en02
ip ospf area 0.0.0.0
ip ospf network point-to-point
ip ospf cost 10
ip ospf authentication message-digest ..
ip ospf message-digest-key ...


ipv6 ospf6 area 0.0.0.0
ipv6 ospf6 network point-to-point
ipv6 ospf6 cost 10
ipv6 ospf6 authentication key-id ..

...