SDN : Mismatch AFI with BGP controller IPv6 session?

[adofou]

Hello,

I try to setup a IPv6 plage into my EVPN zone (shared into 3 PVE in my cluster).
I configure the IPv6 subet on the Vnet and it's working. I can ping the gateway from VM/LXC with this Vnet associted.

My 3 PVEs are exit-node and share prefixes with my network via a BGP controller (to route incoming traffic to the exit-node and enter the EVPN).
I've added an IPv6 session to the BGP controllers and while the GUI accepts it, the configuration generated behind it doesn't seem to take IPv4/IPv6 differences into account.

It's impossible to connect the BGP sessions with my Juniper, because the AFI proposed by PVE is IPv4 unicast when I'm expecting IPv6 unicast.

Dec 18 09:53:36  re1.routeur.par rpd[21588]: BGP_NLRI_MISMATCH: bgp_process_caps: mismatch NLRI with 2001:XXXX:2:103::80:2 (External AS YYYY): peer: <inet-unicast>(1) us: <inet6-unicast>(16)
Dec 18 09:53:36  re1.router.par rpd[21588]: bgp_process_caps:3907: NOTIFICATION sent to 2001:XXXX:103::80:2 (External AS YYYY): code 2 (Open Message Error) subcode 7 (unsupported capability) MP capability afi 1, safi 1 <inet-unicast>

After looking at the FRR configuration, I think it could be that it is placed in the “BGP” group and only seems to be defined in “ipv4 unicast”.
Note : Some particular configurations (route-map on session) come from my frr.conf.local

router bgp YYYY
 bgp router-id X.Y.Z.A
 no bgp default ipv4-unicast
 bgp disable-ebgp-connected-route-check
 coalesce-time 1000
 bgp bestpath as-path multipath-relax
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 10
 neighbor VTEP peer-group
 neighbor VTEP remote-as 31367
 neighbor VTEP bfd
 neighbor VTEP update-source A.B.C.D
 neighbor X.Y.A.IPv4 peer-group BGP
 neighbor X.Y.B.IPv4 peer-group BGP
 neighbor 2001:XXXX:2:103::80:1 peer-group BGP <- IPv6 new session
[...]
 neighbor 172.31.0.2 peer-group VTEP
 neighbor 172.31.0.3 peer-group VTEP
 !
 address-family ipv4 unicast
  network 172.31.0.1/32
  network X.Y.Z.A
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  neighbor BGP allowas-in
  neighbor X.Y.A.IPv4 route-map PUBLIC_RING0_IN in
  neighbor X.Y.B.IPv4 route-map PUBLIC_RING1_IN in
  neighbor 172.31.1.2 route-map EVPN_RING0_IN in
  neighbor 172.31.2.2 route-map EVPN_RING1_IN in
[...]
  import vrf vrf_public
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_public
 exit-address-family
 !

Result :

BGP neighbor is 2001:XXXX:2:103::80:1, remote AS AAAA, local AS YYYY, external link
  Local Role: undefined
  Remote Role: undefined
 Member of peer-group BGP for session parameters
  BGP version 4, remote router ID Y.X.Z.Z, local router ID Y.X.Z.A
  BGP state = Idle
  Last read 18:35:19, Last write 00:01:09
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                563        563
    Notifications:        563          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:               1126        563
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast <- IPv4 Unicast
  BGP peer-group member
  Not part of any update group
  Inbound soft reconfiguration allowed
  Local AS allowed in path, 3 occurrences
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 18:35:19,  Notification sent (OPEN Message Error/Unsupported Capability)
  Message received that caused BGP to send a NOTIFICATION:
    FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
    003F0104 201A005A 53A73F12 22020601
    04000200 01020280 00020202 00020440
    02407802 06410400 00201A02 024700
  External BGP neighbor may be up to 10 hops away.
Local host: 2001:1b48:2:103::80:2, Local port: 38180
Foreign host: 2001:1b48:2:103::80:1, Foreign port: 179
Nexthop: Y.X.Z.Z <- Next-hop IPv4
Nexthop global: 2001:XXXX:2:103::80:2 <- Next-hop IPv6
Nexthop local: fe80::1270:fdff:fe85:b2da
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Next start timer due in 51 seconds
Read thread: off  Write thread: off  FD used: -1
  Capability error for: Multi protocol AFI IPv6, SAFI Unicast

  BFD: Type: multi hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Unknown, Last update: never

I'm not too comfortable with FRR groups yet, so I don't know if adding some "neighbor BGP activate" in unicast IPv6 will make it switch automatically, or if I have to manually create an IPv6 group and match it in unicast ipv6 (that it advertises/receives the right prefixes).

I don't know if it's a limitation of the current design (case not too expected) or if it's a bug.
If anyone has a tip, otherwise I would probably do a BGP6 group manually via my frr.conf.local, hoping it works well with the PVE parser

Thanks!

 

[adofou]

An update : I managed to get my route exchange to work by creating a new group dedicated to IPv6
Note : Of course, I have removed the IPv6 neighbor in the BGP controller in GUI before.

/etc/frr/frr.config.local

router bgp XXXX
 neighbor BGP6 peer-group
 neighbor BGP6 remote-as external
 neighbor BGP6 bfd
 neighbor BGP6 ebgp-multihop 10
 neighbor 2001:XXXX:103::80:1 peer-group BGP6
 address-family ipv6 unicast
  neighbor BGP6 activate
  neighbor BGP6 soft-reconfiguration inbound
 exit-address-family
exit

BGP neighbor is 2001:XXXX:2:103::80:1, remote AS AAAA, local AS YYYY, external link
  Local Role: undefined
  Remote Role: undefined
 Member of peer-group BGP6 for session parameters
  BGP version 4, remote router ID Y.X.Z.Z, local router ID Y.X.Z.A
  BGP state = Established, up for 02:38:02
  Last read 00:00:02, Last write 00:00:02
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised
    AddPath:
      IPv6 Unicast: RX advertised
    Long-lived Graceful Restart: advertised and received
      Address families by peer:
    Route refresh: advertised and received(old & new)
    Enhanced Route Refresh: advertised
    Address Family IPv6 Unicast: advertised and received
    Hostname Capability: advertised (name: prox1,domain name: n/a) not received
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: IPv6 Unicast
    End-of-RIB received: IPv6 Unicast
    Local GR Mode: Helper*

    Remote GR Mode: Helper

    R bit: False
    N bit: True
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
    IPv6 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: No
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                6          2
    Keepalives:          3161       3476
    Route Refresh:          0          0
    Capability:             0          0
    Total:               3168       3479
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv6 Unicast
  BGP6 peer-group member
  Update group 2, subgroup 2
  Packet Queue length 0
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  1 accepted prefixes

  Connections established 1; dropped 0
  Last reset 02:38:06,  Waiting for peer OPEN
  External BGP neighbor may be up to 10 hops away.
Local host: 2001:XXXX:2:103::80:2, Local port: 48074
Foreign host: 2001:XXXX:2:103::80:1, Foreign port: 179
Nexthop: Y.X.Z.Z
Nexthop global: 2001:XXXX:2:103::80:2
Nexthop local: fe80::1270:fdff:fe85:b2da
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Estimated round trip time: 100 ms
Read thread: on  Write thread: on  FD used: 34

  BFD: Type: multi hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Down, Last update: 0:02:38:02

The only funny thing I see is that there's still an IPv4 next-hop and a version of BGP v4 in this show. But at least the family is the right one.
So it seems that it's more a question of Proxmox's configuration generation, which accepts IPv6, but doesn't seem to handle it correctly/isn't aware that it needs to handle it differently.

 

[spirit]

Hi,
for the evpn peers,
currently, we can't haveipv6 vxlan tunnel for evpn (technically possible, but ifupdown2 which manage /etc/network/interfaces don't manage it).

But for bgp peers, I think it's a bug in config generation.

I'l test with splitting ipv4/ipv6 peers group like your configuration, thanks for sharing !

 

[spirit]

I was doing some test on my side, and if I define ipv6 bgp peers, the current code seem also to use ipv6 for vxlan tunnel.

can you check in /etc/network/interfaces.d/sdn ,

the value:

vxlan-local-tunnelip ....


it's should be ipv4 (like the evpn peers), as ipv6 is not supported currently.

 

[spirit]

I have check again, the vxlan-local-tunnelip is ok, only the frr config was wrong


I have sent a fix to the pve-devel mailing list.
https://lists.proxmox.com/pipermail/pve-devel/2024-December/067465.html

if you want to test, I can send you a .deb

 

[adofou]

if you want to test, I can send you a .deb

I can do the test on my own to see if there's any different behavior/side effects between the new generated configuration and the old one+the one that's currently working. Feel free to send me the .deb url in DM or on Discord

Edit : Deleted my next message. I misread the patch with a night of maintenance