second openfabric area with another loopback address

maddig

New Member
Jan 24, 2025
5
2
3
Hi there,

currently i´m trying to setup a second openfabric area. I´m running a 3 node PVE cluster.
I setted up a ceph fully routed mesh network with the offical guide. So far so good, everything is running perfectly.

Every of my 3 nodes have each two additional 10G NICs, which i want to use for cluster network/migration network.

Currently i´m trying this configuration:

Code:
frr defaults traditional
hostname vm003
log syslog warning
ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
interface lo
 ip address 10.254.120.52/32
 ip router openfabric 1
 openfabric passive
!
interface lo:10  <- new additional loopback
 ip router openfabric 2
 openfabric passive
!
interface eno1np0
 ip router openfabric 1
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
interface eno2np1
 ip router openfabric 1
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
interface ens8f0np0 <- NEW additional 10G interface
 ip router openfabric 2
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
interface ens8f1np1 <- NEW additional 10G interface
 ip router openfabric 2
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
line vty
!
router openfabric 1
 net 49.0001.3333.3333.3333.00
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180
!
router openfabric 2 <- NEW additional router
 net 49.0002.3333.3333.3333.00
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180
!

The config ist on all 3 nodes the same except the hostname, router nets and the loopback address.
I tried to configure the loopback in the /etc/network/interfaces like this, because when i configure it in the frr config it is not bonded to the new openfabric router. But both methods are leading to the same result. The lo:10 isnt bonding to the new openfabric router:

Code:
auto lo
iface lo inet loopback

auto lo:10
iface lo:10 inet static
        address 10.254.121.52/32

auto ens4f0np0
iface ens4f0np0 inet manual

auto eno1np0 <- first openfabric
iface eno1np0 inet manual
        mtu 9000

auto eno2np1 <- first openfabric
iface eno2np1 inet manual
        mtu 9000

auto ens4f1np1
iface ens4f1np1 inet manual

auto ens8f0np0 <- second openfabric
iface ens8f0np0 inet manual
        mtu 9000

auto ens8f1np1 <- second openfabric
iface ens8f1np1 inet manual
        mtu 9000

auto bond0
iface bond0 inet manual
        bond-slaves ens4f0np0 ens4f1np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        post-up devlink dev eswitch set pci/0000:af:00.0 mode switchdev
        post-up devlink dev eswitch set pci/0000:af:00.1 mode switchdev

auto mgmt
iface mgmt inet static
        address 10.XXX.99.77/24
        gateway 10.XXX.99.1
        vlan-id 99
        vlan-raw-device vmbr0

post-up /usr/bin/systemctl restart frr.service
source /etc/network/interfaces.d/*

show openfabric interface:
Code:
vm003# show openfabric interface
Area 1:
  Interface   CircId   State    Type     Level
  lo          0x0      Up       loopback L2      
  eno1np0     0x0      Up       p2p      L2      
  eno2np1     0x0      Up       p2p      L2      
Area 2:
  Interface   CircId   State    Type     Level
  ens8f0np0   0x0      Up       p2p      L2      
  ens8f1np1   0x0      Up       p2p      L2      
vm003#

Here it is really clear, that new interface with ip isnt attached to the second openfabric router.

Is there anyway to configure a second loopback for this which is working in the routed network?
Thank you for your help.
 
Last edited:
You can add a dummy interface that carries your router IP and use that in the FRR configuration.
 
Ah, this was the keyword i needed, thank you!

I added a dummy Interface to every host like this:

interfaces:
Code:
auto dummy0
iface dummy0 inet static
    address 10.254.121.52/32
    pre-up ip link add dummy0 type dummy

frr.conf:
Code:
interface dummy0
 ip router openfabric 2
 openfabric passive

Now i see, that the interface is assigend:

Code:
vm003# show openfabric interface
Area 1:
  Interface   CircId   State    Type     Level
  lo          0x0      Up       loopback L2       
  eno1np0     0x0      Up       p2p      L2       
  eno2np1     0x0      Up       p2p      L2       
Area 2:
  Interface   CircId   State    Type     Level
  dummy0      0x0      Up       p2p      L2       
  ens8f0np0   0x0      Up       p2p      L2       
  ens8f1np1   0x0      Up       p2p      L2       
vm003#

But there are only the localhost route:
Code:
vm003# show openfabric route
Area 1:
IS-IS L2 IPv4 routing table:

 Prefix            Metric  Interface  Nexthop        Label(s) 
 --------------------------------------------------------------
 10.254.120.50/32  20      eno1np0    10.254.120.50  -         
 10.254.120.51/32  20      eno2np1    10.254.120.51  -         
 10.254.120.52/32  0       -          -              -         

Area 2:
IS-IS L2 IPv4 routing table:

 Prefix            Metric  Interface  Nexthop  Label(s) 
 --------------------------------------------------------
 10.254.121.52/32  0       -          -        -         

vm003#

and also there are no neighbors for the area2:
Code:
vm003# show openfabric neighbor
Area 1:
  System Id           Interface   L  State        Holdtime SNPA
 vm001               eno1np0     2  Up            2        2020.2020.2020
 vm002               eno2np1     2  Up            2        2020.2020.2020
Area 2:
  System Id           Interface   L  State        Holdtime SNPA
vm003#

I checked, the physical interfaces are up and correct connected, but the dummy0 is in state UNKOWN:
Code:
13: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 7e:95:16:d5:c2:88 brd ff:ff:ff:ff:ff:ff
    inet 10.254.121.52/32 scope global dummy0
       valid_lft forever preferred_lft forever
    inet6 fe80::7c95:16ff:fed5:c288/64 scope link
       valid_lft forever preferred_lft forever

On each host its the same result.

Do i had something done wrong?
 
I've looked into it, I only got it working if I set /31 Ips explicitly on the interfaces belonging to OpenFabric. Then it worked flawlessly with your configuration.
 
Thank you for your help. So you mean /31 IPs directly in /etc/network/interfaces for all physical interfaces on all three nodes?

When every one from the 3 hosts, has two ip, is it possible to select then the migrate network under Cluster->Options?