Mesh Ceph Network with FRR Openfabric - Route question

[cheabred]

Looks like my route setup wants to route over one of the hosts to get to another, even though i have all 3 100G dac cables, attached, it wont direct connect to the other host for some reason?




pve1 => .1
pve2 => .2
pve3 => .3

pve1 port1 -> pve2 port 1
pve1 port 2 -> pve3 port 1
pve3 port2 -> pve2 port 2

first time using FRR at all. i understand the concept, but i dont see why its not realizing there is a less weighted path to get the the .1 server from the


i have switched cables around a ton (waited about 5 min between each move)

Followed this post here (https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Routed_Setup_.28with_Fallback.29)

any help would be appreciated

 

[cbugk]

Your routing tables indicate that for whatever reason link between pve1 and pve2 is ignored. First two comes to mind are bad physical connection, or your FRR configuration could have same host registered as a neighbour twice due to incomplete copy/paste without editing the last bit.

Have you confirmed that your underlay works? That would rule out physical connection issues. Run iperf3 test between pve1 and pve2 but on the underlay IPs.

 

[cheabred]

Here are the 2 FRR configs for those 2 hosts,


there are No ip addresses assigned to the interfaces themselves FRR is handling that.

i did do an iperf test using the FRR ones, and everything is routing fine, except its an extra hop for some reason
tempted to switch to the "Routed Setup (Simple)" method instead of FRR as i feel that would be easier, and also faster switchovers if a cable goes back, or someone for some reason unplugs it.

 

[cbugk]

I am no networking expert and I've come across your post while looking for some insight on CEPH over an SDN for my 5 node cluster.

Having disclaimed that, I got FRR working between 4 nodes (sources listed below) with a routed complete mesh of 10G direct connections. Not that it is relevant, simply a work around for lack of a 10g switch on my part.

So, I had assumed that there needed to be a underlay network for an overlay network to work. I am still firm on that belief. I guess FRR nodes could use IPv6 link-local addresses if they wanted, but that would be a speculation.

Until the cavalry arrives, copy my plan:
* set a routed mesh, ensure each direct link has sufficient throughput
* set FRR nodes on top of that tested underlay
* create the vnet and subnet
* confirm speed test on the VMs

I would first do it on IPv4 for peace of mind and use IPv6 afterwards if that is a requirement.

There on, I, personally, am planning on keeping Ceph on the routed network and using that network for backup cluster communication as well. It is the internal network and needn't be modified, that allows for complete remake of SDN for my experiments. But, you do you.

I am going to prepare a blog for this, and might update my route according to your experience, so any confirmation is welcome. Here is my blog for routed mesh based on official docs, just another source to compare: https://blog.cbugk.com/post/proxmox-ceph-routed-ipv6/.

And below are what made me arrive on a working setup of FRR on 10G mesh. Unfortunately I don't have its blog ready:
* https://vincent.bernat.ch/en/blog/2017-vxlan-bgp-evpn
* https://bennetgallein.de/blog/setting-up-evpn-on-proxmox-sdn-a-comprehensive-guide
* https://myarbitrarystuff.com/2019/03/02/programmable-fabric-vxlan-bgp-evpn-part-1/

Edit: Oh the neighbour I was talking about is from bennetgailein link:

router bgp 6500
...
neighbor 172.16.0.2 peer-group VTEP
neighbor 172.16.0.3 peer-group VTEP

Which proves your point of underlay not being part of the FRR config. Official docs shows that correlation is set on the EVPN definition as in:

ID: myevpnctl
ASN#: 65000
Peers: 192.168.0.1,192.168.0.2,192.168.0.3

I need some testing to provide actual useful information it seems.