SDN Fabric issues since update 9.1.14

F

Flocki

New Member
Sep 1, 2025
8
0
1
Hi,

My setup is as follows:
Cluster of 3 x Minisforum MS-01, connected via a SDN Fabric over Thunderbolt.

This setup has worked without problems for several month.

Last week I updated from 9.1.9 to 9.1.14, after which SDN prompted me to re-push the SDN config due to deprecated settings.
After that push traffic over the fabric failed.
On vtysh I could see that the underlying fabric still was built but I could not connect between any of the nodes (SSH, Ping, ...)
I realized that the previous setup had put every node in its own /32 subnet, which was working fine before the update but not afterwards.
I used the new matrix on the fabric tab to change all nodes into /24 subnets (there was no automatic migration the matrix was completely empty and it "IP"-field on top does not take Subnets)
Since then connections over the fabric do not start reliable. Sometimes everything just works fins, but sometimes connection fail, which I could track down to ARP issues. The ARP-Table then shows "(incomplete)" for one or both of the other nodes. Typically this state is then seen on both ends of the connection.
Repeatably restarting the interfaces (ifdown/ifup) will finally lead to a working connection, but might need several tries.

To make sure the Thunderbolt-net driver is not the causer I pinned an old known-good Kernel, but that did not change anything.

Any ideas on how to get the fabric back in a reliable state are welcome.

Florian
 
Please post the output of

Code: 
cat /etc/pve/sdn/fabrics.cfg
ip a
ip r
vtysh -c 'show openfabric interface'
vtysh -c 'show openfabric neighbor'
vtysh -c 'show openfabric summary'

on one node
 
Code: 
ip r


default via 192.168.127.254 dev vmbr0 proto kernel onlink

192.168.125.0/24 dev en05 proto kernel scope link src 192.168.125.11

192.168.125.0/24 dev en06 proto kernel scope link src 192.168.125.11

192.168.125.12 nhid 272 via 192.168.125.12 dev en06 proto openfabric src 192.168.125.11 metric 20 onlink

192.168.125.13 nhid 181 via 192.168.125.13 dev en05 proto openfabric src 192.168.125.11 metric 20 onlink

192.168.127.0/24 dev vmbr0 proto kernel scope link src 192.168.127.11

Code: 
cat /etc/pve/sdn/fabrics.cfg


openfabric_fabric: backhaul

        csnp_interval 2

        hello_interval 1

        ip_prefix 192.168.125.0/24


openfabric_node: backhaul_pmxnode01

        interfaces name=en05,ip=192.168.125.11/24

        interfaces name=en06,ip=192.168.125.11/24

        ip 192.168.125.11


openfabric_node: backhaul_pmxnode02

        interfaces name=en05,ip=192.168.125.12/24

        interfaces name=en06,ip=192.168.125.12/24

        ip 192.168.125.12


openfabric_node: backhaul_pmxnode03

        interfaces name=en06,ip=192.168.125.13/24

        interfaces name=en05,ip=192.168.125.13/24

        ip 192.168.125.13


Code: 
vtysh -c 'show openfabric interface'


Area backhaul:

  Interface   CircId   State    Type     Level

  dummy_backhaul0x0      Up       loopback L2       

  en05        0x0      Up       p2p      L2       

  en06        0x0      Up       p2p      L2


Code: 
vtysh -c 'show openfabric neighbor'


Area backhaul:

 System Id           Interface   L  State         Holdtime SNPA

 pmxnode03           en05        2  Up            9        2020.2020.2020

 pmxnode02           en06        2  Up            10       2020.2020.2020


Code: 
vtysh -c 'show openfabric summary'


vrf             : default

Process Id      : 2033

System Id       : 1921.6812.5011

Up time         : 03:12:10 ago

Number of areas : 1

Area backhaul:

  Tier: undefined

  Net: 49.0001.1921.6812.5011.00

  TX counters per PDU type:

    P2P IIH: 24116

     L2 LSP: 5

     FS LSP: 59

    L2 CSNP: 22858

    L2 PSNP: 64

   LSP RXMT: 0

  RX counters per PDU type:

    P2P IIH: 24105

     L2 LSP: 6

     FS LSP: 66

    L2 CSNP: 22856

    L2 PSNP: 55

  Drop counters per PDU type:

  Advertise high metrics: Disabled

  Level-2:

    LSP0 regenerated: 35

         LSPs purged: 0

    SPF:

      minimum interval  : 1

    IPv4 route computation:

      last run elapsed  : 00:09:40 ago

      last run duration : 127 usec

      run count         : 102


Code: 
cat /etc/frr/frr.conf


frr version 10.6.1

frr defaults datacenter

hostname pmxnode01

log syslog informational

service integrated-vtysh-config

!

router openfabric backhaul

 net 49.0001.1921.6812.5011.00

exit

!

interface dummy_backhaul

 ip router openfabric backhaul

 openfabric passive

exit

!

interface en05

 ip router openfabric backhaul

 openfabric hello-interval 1

 openfabric csnp-interval 2

exit

!

interface en06

 ip router openfabric backhaul

 openfabric hello-interval 1

 openfabric csnp-interval 2

exit

!

access-list pve_openfabric_backhaul_ips permit 192.168.125.0/24

!

route-map pve_openfabric permit 100

 match ip address pve_openfabric_backhaul_ips

 set src 192.168.125.11

exit

!

ip protocol openfabric route-map pve_openfabric

!

line vty

!
 
Code: 
ip a


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host noprefixroute

       valid_lft forever preferred_lft forever

2: nic0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:37 brd ff:ff:ff:ff:ff:ff

    altname enx5847ca7c9137

3: nic1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:39 brd ff:ff:ff:ff:ff:ff permaddr 58:47:ca:7c:91:38

    altname enx5847ca7c9138

4: nic2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    altname enx5847ca7c9135

5: nic3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff permaddr 58:47:ca:7c:91:36

    altname enx5847ca7c9136

8: nic4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

    link/ether 78:93:c3:bb:0c:08 brd ff:ff:ff:ff:ff:ff

9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:39 brd ff:ff:ff:ff:ff:ff

    inet 192.168.127.11/24 scope global vmbr0

       valid_lft forever preferred_lft forever

    inet6 fd50:6f7d:70e5:807f::b/64 scope global

       valid_lft forever preferred_lft forever

    inet6 fe80::5a47:caff:fe7c:9139/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

11: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:37 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9137/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

13: ln_vlan0016@pr_vlan0016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0016 state UP group default qlen 1000

    link/ether a2:65:cb:09:6c:a6 brd ff:ff:ff:ff:ff:ff

14: pr_vlan0016@ln_vlan0016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v16 state UP group default qlen 1000

    link/ether 0e:f6:cc:44:52:fd brd ff:ff:ff:ff:ff:ff

15: vlan0016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether a2:65:cb:09:6c:a6 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::a065:cbff:fe09:6ca6/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

16: ln_vlan0032@pr_vlan0032: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0032 state UP group default qlen 1000

    link/ether 5e:4f:94:f1:60:1c brd ff:ff:ff:ff:ff:ff

17: pr_vlan0032@ln_vlan0032: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v32 state UP group default qlen 1000

    link/ether ba:d1:24:e0:c7:49 brd ff:ff:ff:ff:ff:ff

18: vlan0032: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 5e:4f:94:f1:60:1c brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5c4f:94ff:fef1:601c/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

19: ln_vlan0048@pr_vlan0048: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0048 state UP group default qlen 1000

    link/ether f6:8a:25:91:c3:85 brd ff:ff:ff:ff:ff:ff

20: pr_vlan0048@ln_vlan0048: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v48 state UP group default qlen 1000

    link/ether b6:8a:16:4f:dc:a2 brd ff:ff:ff:ff:ff:ff

21: vlan0048: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether f6:8a:25:91:c3:85 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::f48a:25ff:fe91:c385/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

22: ln_vlan0064@pr_vlan0064: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0064 state UP group default qlen 1000

    link/ether 8a:af:d6:3a:be:48 brd ff:ff:ff:ff:ff:ff

23: pr_vlan0064@ln_vlan0064: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v64 state UP group default qlen 1000

    link/ether fa:bb:78:e4:32:80 brd ff:ff:ff:ff:ff:ff

24: vlan0064: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 8a:af:d6:3a:be:48 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::88af:d6ff:fe3a:be48/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

25: ln_vlan0072@pr_vlan0072: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0072 state UP group default qlen 1000

    link/ether 6a:48:3a:56:19:48 brd ff:ff:ff:ff:ff:ff

26: pr_vlan0072@ln_vlan0072: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v72 state UP group default qlen 1000

    link/ether ca:9b:f3:7f:74:4c brd ff:ff:ff:ff:ff:ff

27: vlan0072: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 6a:48:3a:56:19:48 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::6848:3aff:fe56:1948/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

28: ln_vlan0073@pr_vlan0073: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0073 state UP group default qlen 1000

    link/ether 9e:16:36:4e:88:2b brd ff:ff:ff:ff:ff:ff

29: pr_vlan0073@ln_vlan0073: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v73 state UP group default qlen 1000

    link/ether f2:2f:d3:12:f7:a3 brd ff:ff:ff:ff:ff:ff

30: vlan0073: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 9e:16:36:4e:88:2b brd ff:ff:ff:ff:ff:ff

    inet6 fe80::9c16:36ff:fe4e:882b/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

31: ln_vlan0074@pr_vlan0074: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0074 state UP group default qlen 1000

    link/ether 7e:e4:a4:e2:9e:63 brd ff:ff:ff:ff:ff:ff

32: pr_vlan0074@ln_vlan0074: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v74 state UP group default qlen 1000

    link/ether a2:4f:29:07:d9:eb brd ff:ff:ff:ff:ff:ff

33: vlan0074: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 7e:e4:a4:e2:9e:63 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::7ce4:a4ff:fee2:9e63/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

34: ln_vlan0080@pr_vlan0080: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0080 state UP group default qlen 1000

    link/ether 8a:a7:80:a5:ec:57 brd ff:ff:ff:ff:ff:ff

35: pr_vlan0080@ln_vlan0080: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr2v80 state UP group default qlen 1000

    link/ether 32:24:82:4e:55:c1 brd ff:ff:ff:ff:ff:ff

36: vlan0080: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 8a:a7:80:a5:ec:57 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::88a7:80ff:fea5:ec57/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

37: ln_vlan0126@pr_vlan0126: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0126 state UP group default qlen 1000

    link/ether 1a:84:ff:c3:94:18 brd ff:ff:ff:ff:ff:ff

38: pr_vlan0126@ln_vlan0126: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v126 state UP group default qlen 1000

    link/ether ea:e8:34:77:b3:36 brd ff:ff:ff:ff:ff:ff

39: vlan0126: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 1a:84:ff:c3:94:18 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::1884:ffff:fec3:9418/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

40: bond0.126@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v126 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

41: vmbr1v126: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

42: bond0.16@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v16 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

43: vmbr1v16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

44: bond0.32@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v32 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

45: vmbr1v32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

46: bond0.48@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v48 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

47: vmbr1v48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

48: bond0.64@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v64 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

49: vmbr1v64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

50: bond0.72@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v72 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

51: vmbr1v72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

52: bond0.73@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v73 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

53: vmbr1v73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

54: bond0.74@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v74 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

55: vmbr1v74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:35 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9135/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

56: nic0.80@nic0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr2v80 state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:37 brd ff:ff:ff:ff:ff:ff

57: vmbr2v80: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 58:47:ca:7c:91:37 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::5a47:caff:fe7c:9137/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

58: dummy_backhaul: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether 36:e4:a2:9a:a2:57 brd ff:ff:ff:ff:ff:ff

    inet 192.168.125.11/32 scope global dummy_backhaul

       valid_lft forever preferred_lft forever

    inet6 fe80::34e4:a2ff:fe9a:a257/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

59: tap1107i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vmbr0 state UNKNOWN group default qlen 1000

    link/ether 32:4b:f1:5f:48:7c brd ff:ff:ff:ff:ff:ff

60: tap1505i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vlan0064 state UNKNOWN group default qlen 1000

    link/ether d6:12:f2:d7:0f:4f brd ff:ff:ff:ff:ff:ff

61: tap1105i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UNKNOWN group default qlen 1000

    link/ether 0e:79:a3:62:19:91 brd ff:ff:ff:ff:ff:ff

62: tap1103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UNKNOWN group default qlen 1000

    link/ether 0a:05:7d:6a:6b:86 brd ff:ff:ff:ff:ff:ff

63: tap1101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UNKNOWN group default qlen 1000

    link/ether 8e:53:22:a9:1e:20 brd ff:ff:ff:ff:ff:ff

64: tap1103i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0072 state UNKNOWN group default qlen 1000

    link/ether 82:a1:fe:9c:73:0c brd ff:ff:ff:ff:ff:ff

65: tap1105i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0072 state UNKNOWN group default qlen 1000

    link/ether de:4a:8b:56:fe:6d brd ff:ff:ff:ff:ff:ff

66: tap1101i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0064 state UNKNOWN group default qlen 1000

    link/ether 82:d9:16:39:a4:ef brd ff:ff:ff:ff:ff:ff

67: tap1103i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vlan0126 state UNKNOWN group default qlen 1000

    link/ether 56:93:08:d9:f3:ba brd ff:ff:ff:ff:ff:ff

68: tap1105i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vlan0126 state UNKNOWN group default qlen 1000

    link/ether a2:66:8a:4d:62:90 brd ff:ff:ff:ff:ff:ff

69: tap1101i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0072 state UNKNOWN group default qlen 1000

    link/ether 2a:2d:b0:c2:23:83 brd ff:ff:ff:ff:ff:ff
 
Code: 
ip a (continued)

70: tap1103i3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0073 state UNKNOWN group default qlen 1000

    link/ether 8a:fb:d5:86:91:d3 brd ff:ff:ff:ff:ff:ff

71: tap1105i3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0074 state UNKNOWN group default qlen 1000

    link/ether d2:7a:c2:24:db:6a brd ff:ff:ff:ff:ff:ff

72: tap1101i3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vlan0126 state UNKNOWN group default qlen 1000

    link/ether 1a:23:22:f6:07:f8 brd ff:ff:ff:ff:ff:ff

73: tap1101i4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vlan0080 state UNKNOWN group default qlen 1000

    link/ether 52:cc:b8:38:97:35 brd ff:ff:ff:ff:ff:ff

74: tap1101i5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0016 state UNKNOWN group default qlen 1000

    link/ether ee:ee:7a:df:fc:d9 brd ff:ff:ff:ff:ff:ff

75: tap1101i6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0032 state UNKNOWN group default qlen 1000

    link/ether 6e:f8:b8:25:e1:7b brd ff:ff:ff:ff:ff:ff

76: tap1101i7: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0048 state UNKNOWN group default qlen 1000

    link/ether 0e:a5:af:3a:92:6f brd ff:ff:ff:ff:ff:ff

77: tap1301i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master fwbr1301i0 state UNKNOWN group default qlen 1000

    link/ether 42:89:4d:44:98:56 brd ff:ff:ff:ff:ff:ff

78: fwbr1301i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 02:9e:ad:37:f4:a0 brd ff:ff:ff:ff:ff:ff

79: fwpr1301p0@fwln1301i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan0064 state UP group default qlen 1000

    link/ether f2:82:f4:4e:05:59 brd ff:ff:ff:ff:ff:ff

80: fwln1301i0@fwpr1301p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr1301i0 state UP group default qlen 1000

    link/ether 02:9e:ad:37:f4:a0 brd ff:ff:ff:ff:ff:ff

81: tap1504i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vlan0064 state UNKNOWN group default qlen 1000

    link/ether 5e:77:f1:bc:98:af brd ff:ff:ff:ff:ff:ff

87: en05: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 1000

    link/ether 02:db:47:6e:e1:fa brd ff:ff:ff:ff:ff:ff

    inet 192.168.125.11/24 scope global en05

       valid_lft forever preferred_lft forever

    inet6 fe80::db:47ff:fe6e:e1fa/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever

92: en06: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 1000

    link/ether 02:53:97:a8:f6:94 brd ff:ff:ff:ff:ff:ff

    inet 192.168.125.11/24 scope global en06

       valid_lft forever preferred_lft forever

    inet6 fe80::53:97ff:fea8:f694/64 scope link proto kernel_ll

       valid_lft forever preferred_lft forever
 
Can you try removing the 192.168.125.X/24 that you configured on the interfaces themselves (en05, en06) - then reapply the configuration? The /32 are auto-generated by our stack for unnumbered OpenFabric fabrics, since it prevents the kernel from selecting the wrong source IP for ARP packets.

In case that doesn't work, please provide the output of the following commands again:

Code: 
ip r
vtysh -c 'show openfabric interface'
vtysh -c 'show openfabric neighbor'
vtysh -c 'show openfabric summary'
cat /etc/frr/frr.conf
 
Removing the entries completely removes any connectivity between the nodes.

Code: 
ip r

default via 192.168.127.254 dev vmbr0 proto kernel onlink
192.168.125.12 nhid 107 via 192.168.125.12 dev en06 proto openfabric src 192.168.125.11 metric 20 onlink
192.168.125.13 nhid 108 via 192.168.125.13 dev en05 proto openfabric src 192.168.125.11 metric 20 onlink
192.168.127.0/24 dev vmbr0 proto kernel scope link src 192.168.127.11

Code: 
vtysh -c 'show openfabric interface'

Area backhaul:
  Interface   CircId   State    Type     Level
  dummy_backhaul0x0      Up       loopback L2       
  en05        0x0      Up       p2p      L2       
  en06        0x0      Up       p2p      L2

Code: 
vtysh -c 'show openfabric neighbor'

Area backhaul:
 System Id           Interface   L  State         Holdtime SNPA
 pmxnode03           en05        2  Up            10       2020.2020.2020
 pmxnode02           en06        2  Up            10       2020.2020.2020

Code: 
vtysh -c 'show openfabric summary'

vrf             : default
Process Id      : 2034
System Id       : 1921.6812.5011
Up time         : 00:03:40 ago
Number of areas : 1
Area backhaul:
  Tier: undefined
  Net: 49.0001.1921.6812.5011.00
  TX counters per PDU type:
    P2P IIH: 468
     L2 LSP: 1
     FS LSP: 4
    L2 CSNP: 432
    L2 PSNP: 3
   LSP RXMT: 0
  RX counters per PDU type:
    P2P IIH: 464
     L2 LSP: 1
     FS LSP: 3
    L2 CSNP: 433
    L2 PSNP: 3
  Drop counters per PDU type:
  Advertise high metrics: Disabled
  Level-2:
    LSP0 regenerated: 3
         LSPs purged: 0
    SPF:
      minimum interval  : 1
    IPv4 route computation:
      last run elapsed  : 00:03:15 ago
      last run duration : 50 usec
      run count         : 5

Code: 
cat /etc/frr/frr.conf

frr version 10.6.1
frr defaults datacenter
hostname pmxnode01
log syslog informational
service integrated-vtysh-config
!
router openfabric backhaul
 net 49.0001.1921.6812.5011.00
exit
!
interface dummy_backhaul
 ip router openfabric backhaul
 openfabric passive
exit
!
interface en05
 ip router openfabric backhaul
 openfabric hello-interval 1
 openfabric csnp-interval 2
exit
!
interface en06
 ip router openfabric backhaul
 openfabric hello-interval 1
 openfabric csnp-interval 2
exit
!
access-list pve_openfabric_backhaul_ips permit 192.168.125.0/24
!
route-map pve_openfabric permit 100
 match ip address pve_openfabric_backhaul_ips
 set src 192.168.125.11
exit
!
ip protocol openfabric route-map pve_openfabric
!
line vty
!

Code: 
arp

Address                  HWtype  HWaddress           Flags Mask            Iface
pmxnode02.laufenstein.a  ether   58:47:ca:7c:66:c2   C                     vmbr0
192.168.125.12                   (incomplete)                              en06
192.168.127.254          ether   00:00:5e:00:01:10   C                     vmbr0
192.168.125.13                   (incomplete)                              en05
pmxnode03.laufenstein.a  ether   58:47:ca:7c:5d:52   C                     vmbr0
 
Flocki said:
Removing the entries completely removes any connectivity between the nodes.
Click to expand...
From the routing table entries, as well as the vtysh output, it looks as if FRR is able to find the neighbors and learn / install the proper routes.

How does the interface configuration look like?
Code: 
ip a show dummy_backhaul
ip a show en05
ip a show en06

Can you run a tcpdump on two nodes while trying to ping from one node to another?

Code: 
tcpdump -envi any arp or icmp
 
Hi,

this is the configuration and the tcpdump is attached as files.

Node 1:

Code: 
ip a show dummy_backhaul

58: dummy_backhaul: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 46:57:f4:c1:e8:af brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.11/32 scope global dummy_backhaul
       valid_lft forever preferred_lft forever
    inet6 fe80::4457:f4ff:fec1:e8af/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
Code: 
ip a show en05
      
6: en05: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:55:52:f8:09:fd brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.11/32 scope global en05
       valid_lft forever preferred_lft forever
    inet6 fe80::55:52ff:fef8:9fd/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
Code: 
ip a show en06       

7: en06: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:2d:b3:f4:c0:4e brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.11/32 scope global en06
       valid_lft forever preferred_lft forever
    inet6 fe80::2d:b3ff:fef4:c04e/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
Code: 
ping 192.168.125.12 (Node 1 -> Node 2)

PING 192.168.125.12 (192.168.125.12) 56(84) bytes of data.
From 192.168.125.11 icmp_seq=1 Destination Host Unreachable
From 192.168.125.11 icmp_seq=2 Destination Host Unreachable
From 192.168.125.11 icmp_seq=3 Destination Host Unreachable
From 192.168.125.11 icmp_seq=8 Destination Host Unreachable
From 192.168.125.11 icmp_seq=9 Destination Host Unreachable
Code: 
arp

Address                  HWtype  HWaddress           Flags Mask            Iface
opnmain01.laufenstein.a  ether   bc:24:11:27:8b:41   C                     vmbr0
pmxnode02.laufenstein.a  ether   58:47:ca:7c:66:c2   C                     vmbr0
192.168.125.12                   (incomplete)                              en06
192.168.127.254          ether   00:00:5e:00:01:10   C                     vmbr0
192.168.125.13                   (incomplete)                              en05
pmxnode03.laufenstein.a  ether   58:47:ca:7c:5d:52   C                     vmbr0
192.168.127.244          ether   bc:24:11:7f:9b:ea   C                     vmbr0

Node 2:

Code: 
ip a show dummy_backhaul

58: dummy_backhaul: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 16:1b:7b:65:ad:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.12/32 scope global dummy_backhaul
       valid_lft forever preferred_lft forever
    inet6 fe80::141b:7bff:fe65:add8/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
Code: 
ip a show en05

7: en05: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:0e:4c:5b:b8:6c brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.12/32 scope global en05
       valid_lft forever preferred_lft forever
    inet6 fe80::e:4cff:fe5b:b86c/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
Code: 
ip a show en06

86: en06: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:b3:6a:9d:58:92 brd ff:ff:ff:ff:ff:ff
    inet 192.168.125.12/32 scope global en06
       valid_lft forever preferred_lft forever
    inet6 fe80::b3:6aff:fe9d:5892/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
Code: 
arp

192.168.125.11                   (incomplete)                              en06
192.168.125.13                   (incomplete)                              en05
192.168.127.254          ether   00:00:5e:00:01:10   C                     vmbr0
pmxnode03.laufenstein.a  ether   58:47:ca:7c:5d:52   C                     vmbr0
pmxnode01.laufenstein.a  ether   58:47:ca:7c:91:39   C                     vmbr0
192.168.127.244          ether   bc:24:11:7f:9b:ea   C                     vmbr0
opnmain01.laufenstein.a  ether   bc:24:11:27:8b:41   C                     vmbr0

Thanks, Florian
 

Attachments

You must log in or register to reply here.
Top Bottom
Back