second openfabric area with another loopback address

M

maddig

New Member
Jan 24, 2025
6
2
3
Hi there,

currently i´m trying to setup a second openfabric area. I´m running a 3 node PVE cluster.
I setted up a ceph fully routed mesh network with the offical guide. So far so good, everything is running perfectly.

Every of my 3 nodes have each two additional 10G NICs, which i want to use for cluster network/migration network.

Currently i´m trying this configuration:

Code: 
frr defaults traditional
hostname vm003
log syslog warning
ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
interface lo
 ip address 10.254.120.52/32
 ip router openfabric 1
 openfabric passive
!
interface lo:10  <- new additional loopback
 ip router openfabric 2
 openfabric passive
!
interface eno1np0
 ip router openfabric 1
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
interface eno2np1
 ip router openfabric 1
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
interface ens8f0np0 <- NEW additional 10G interface
 ip router openfabric 2
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
interface ens8f1np1 <- NEW additional 10G interface
 ip router openfabric 2
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
line vty
!
router openfabric 1
 net 49.0001.3333.3333.3333.00
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180
!
router openfabric 2 <- NEW additional router
 net 49.0002.3333.3333.3333.00
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180
!

The config ist on all 3 nodes the same except the hostname, router nets and the loopback address.
I tried to configure the loopback in the /etc/network/interfaces like this, because when i configure it in the frr config it is not bonded to the new openfabric router. But both methods are leading to the same result. The lo:10 isnt bonding to the new openfabric router:

Code: 
auto lo
iface lo inet loopback

auto lo:10
iface lo:10 inet static
        address 10.254.121.52/32

auto ens4f0np0
iface ens4f0np0 inet manual

auto eno1np0 <- first openfabric
iface eno1np0 inet manual
        mtu 9000

auto eno2np1 <- first openfabric
iface eno2np1 inet manual
        mtu 9000

auto ens4f1np1
iface ens4f1np1 inet manual

auto ens8f0np0 <- second openfabric
iface ens8f0np0 inet manual
        mtu 9000

auto ens8f1np1 <- second openfabric
iface ens8f1np1 inet manual
        mtu 9000

auto bond0
iface bond0 inet manual
        bond-slaves ens4f0np0 ens4f1np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        post-up devlink dev eswitch set pci/0000:af:00.0 mode switchdev
        post-up devlink dev eswitch set pci/0000:af:00.1 mode switchdev

auto mgmt
iface mgmt inet static
        address 10.XXX.99.77/24
        gateway 10.XXX.99.1
        vlan-id 99
        vlan-raw-device vmbr0

post-up /usr/bin/systemctl restart frr.service
source /etc/network/interfaces.d/*

show openfabric interface:
Code: 
vm003# show openfabric interface
Area 1:
  Interface   CircId   State    Type     Level
  lo          0x0      Up       loopback L2      
  eno1np0     0x0      Up       p2p      L2      
  eno2np1     0x0      Up       p2p      L2      
Area 2:
  Interface   CircId   State    Type     Level
  ens8f0np0   0x0      Up       p2p      L2      
  ens8f1np1   0x0      Up       p2p      L2      
vm003#

Here it is really clear, that new interface with ip isnt attached to the second openfabric router.

Is there anyway to configure a second loopback for this which is working in the routed network?
Thank you for your help.
 
Last edited:
You can add a dummy interface that carries your router IP and use that in the FRR configuration.
 
Ah, this was the keyword i needed, thank you!

I added a dummy Interface to every host like this:

interfaces:
Code: 
auto dummy0
iface dummy0 inet static
    address 10.254.121.52/32
    pre-up ip link add dummy0 type dummy

frr.conf:
Code: 
interface dummy0
 ip router openfabric 2
 openfabric passive

Now i see, that the interface is assigend:

Code: 
vm003# show openfabric interface
Area 1:
  Interface   CircId   State    Type     Level
  lo          0x0      Up       loopback L2       
  eno1np0     0x0      Up       p2p      L2       
  eno2np1     0x0      Up       p2p      L2       
Area 2:
  Interface   CircId   State    Type     Level
  dummy0      0x0      Up       p2p      L2       
  ens8f0np0   0x0      Up       p2p      L2       
  ens8f1np1   0x0      Up       p2p      L2       
vm003#

But there are only the localhost route:
Code: 
vm003# show openfabric route
Area 1:
IS-IS L2 IPv4 routing table:

 Prefix            Metric  Interface  Nexthop        Label(s) 
 --------------------------------------------------------------
 10.254.120.50/32  20      eno1np0    10.254.120.50  -         
 10.254.120.51/32  20      eno2np1    10.254.120.51  -         
 10.254.120.52/32  0       -          -              -         

Area 2:
IS-IS L2 IPv4 routing table:

 Prefix            Metric  Interface  Nexthop  Label(s) 
 --------------------------------------------------------
 10.254.121.52/32  0       -          -        -         

vm003#

and also there are no neighbors for the area2:
Code: 
vm003# show openfabric neighbor
Area 1:
  System Id           Interface   L  State        Holdtime SNPA
 vm001               eno1np0     2  Up            2        2020.2020.2020
 vm002               eno2np1     2  Up            2        2020.2020.2020
Area 2:
  System Id           Interface   L  State        Holdtime SNPA
vm003#

I checked, the physical interfaces are up and correct connected, but the dummy0 is in state UNKOWN:
Code: 
13: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 7e:95:16:d5:c2:88 brd ff:ff:ff:ff:ff:ff
    inet 10.254.121.52/32 scope global dummy0
       valid_lft forever preferred_lft forever
    inet6 fe80::7c95:16ff:fed5:c288/64 scope link
       valid_lft forever preferred_lft forever

On each host its the same result.

Do i had something done wrong?
 
I've looked into it, I only got it working if I set /31 Ips explicitly on the interfaces belonging to OpenFabric. Then it worked flawlessly with your configuration.
 
Thank you for your help. So you mean /31 IPs directly in /etc/network/interfaces for all physical interfaces on all three nodes?

When every one from the 3 hosts, has two ip, is it possible to select then the migrate network under Cluster->Options?
 
maddig said:
Thank you for your help. So you mean /31 IPs directly in /etc/network/interfaces for all physical interfaces on all three nodes?
Click to expand...

Yes, use the physical interfaces to configure point-to-point connections and configure a dummy interface that carries the router IP in the fabric. Since the dummy interface will have a /32 IP, you will need to manually edit the datacenter.cfg, otherwise the migration task won't pick up on the IPs of the other nodes. I use 172.21.255.0/24 for all my router IPs, so I configure this CIDR as migration network:

Code: 
migration: network=172.21.255.0/24,type=secure
 
Thank you!! I had to clarify it once again because i´m a little bit affraid because my ceph storage is also running over ffr and the last time i played arround and restarted the service, my ceph network was halfway gone...

example config on first host:

interfaces
Code: 
auto dummy0
iface dummy0 inet static
    pre-up ip link add dummy0 type dummy

auto ens8f0np0
iface ens8f0np0 inet static
    address 10.254.121.2/31
    mtu 9000

auto ens8f1np1
iface ens8f1np1 inet static
    address 10.254.121.3/31
    mtu 9000

.4&.5 and .6&.7 /31 for the other 2 hosts

frr.conf
Code: 
interface dummy0
 ip address 10.254.121.50/32  ---- .51 & .52 for the other 2 hosts
 ip router openfabric 2
 openfabric passive
 
 interface ens8f0np0
 ip router openfabric 2
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
interface ens8f1np1
 ip router openfabric 2
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
 
 router openfabric 2
 net 49.0002.1111.1111.1111.00
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180

is this config correct?

i will edit the datacenter.cfg manually, but how could i apply your patch? Sorry for the question, i´m not into in this dev stuff :(

Thank you for your help!
 
maddig said:
and also there are no neighbors for the area2:
Click to expand...
after you've applied that config, could you do a `wt t` and paste the output here?

AFAIK The Openfabric in FRRouting (which needs to die a death but some stuff needs to be ported to the IS-IS first when you chat to Russ White) and only the IS-IS part supports multiple areas AFAIK but needs to be in different VRFs if you need them to be separated
 
You might want to try the new SDN Fabrics feature, where we took care that creating multiple Openfabric areas is possible. It is available in the BETA release, and will stay in a preview state after the PVE 9 release, but it should work fairly well for your usecase.
 
Hmmm...haven't been able to make it work on FRR 10.3 nor 10.4 so that is why I'm asking for the running `wr t` output of one that is configured on.
 
Can you try setting the router id the same for both routers? IIRC FRR doesn't handle multiple router-ids (even with multiple OpenFabric routers) very well.
 
You must log in or register to reply here.
Top Bottom