[TUTORIAL] [Full mesh (routed setup) + EVPN] it is feasible even by using SDN!

interesting tutorial - i am hoping this can help me expose my thunderbolt mesh network (used for ceph public and cluster).

i have 3 node thunderbolt mesh based on this guide
it works great

this is the existing openfabric routing on each promox node, where X is the node number.
(note i only use the fc00::x/128 addresses for the ceph network, the IPv4 is there just because)

Code:
ip forwarding
ipv6 forwarding
!
interface en05
ip router openfabric 1
ipv6 router openfabric 1
exit
!
interface en06
ip router openfabric 1
ipv6 router openfabric 1
exit
!
interface lo
ip address 10.0.0.8x/32
ipv6 address fc00::8x/128
ip router openfabric 1
ipv6 router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0000.0000.000x.00
exit
!
exit
this is currently saved in both frr.conf and frr.local

this the global section of my ceph cluster
Code:
[global]
        auth_client_required = cephx
        auth_cluster_required = cephx
        auth_service_required = cephx
        cluster_network = fc00::/64
        fsid = 5e55fd50-d135-413d-bffe-9d0fae0ef5fa
        mon_allow_pool_delete = true
        mon_host = fc00::83 fc00::82 fc00::81
        ms_bind_ipv4 = false
        ms_bind_ipv6 = true
        osd_pool_default_min_size = 2
        osd_pool_default_size = 3
        public_network = fc00::/64

what i would like to achieve is:
  1. keep the ceph public network on the thunderbolt - i want the proxmox QEMU processes to use that route and network (for example in failure scenarios its better to have a node connect to another node across the tunderbolt network)
  2. enable VMs (with existing globally routable IPv6 addresses) on the proxmox node to access the mesh network (fc00::81 thru fc00::83)
  3. enable clients on the LAN (with existing globally routable IPv6 addresses) to access the mesh network addresses (fc00::81 thru fc00::83)
questions:
  1. can i just follow the guide as-is?
  2. will the existing openfabric routing co-exist with the new bgp based routing or will the new bgp routing replace the openfabric routing?
  3. i see the options for exit nodes and advertising - i assume i need to use these in some way? (but i am unsure how they would be configured)
  4. my unifi router is capable of frr under the hood and bgp in the UI - i assume that can assist in providing route information to the clilents, but i am, unsure of the right way to go about this?
  5. is there a way to apply the SDN to one node at a time - i.e. to be able to do any frr.conf fix ups as i go along and keep my ceph running?
  6. any other gotchas i need to think about?
 
Last edited: