[SOLVED] BGP Controller breaks EVPN

G

geforce

New Member
Aug 3, 2025
4
0
1
Canada
Hello everyone,

I have a newly created 3 nodes pve cluster (v8.4). SDN works fine out of the box: I created an EVPN controller and zone, and the vnets work straight away.
However, as soon as I add a BGP controller, it breaks the EVPN: the node that establish a BGP connection with my router lose it's iBGP EVPN peers.


Has anyone encountered that?


Thanks!

Here is the running-config on the pve node with the added BGP controller:

Code: 
# show running-config
Building configuration...

Current configuration:
!
frr version 10.2.2
frr defaults datacenter
hostname pve03
log syslog informational
service integrated-vtysh-config
!
ip prefix-list loopbacks_ips seq 10 permit 0.0.0.0/0 le 32
ip prefix-list only_default seq 1 permit 0.0.0.0/0
!
ipv6 prefix-list only_default_v6 seq 1 permit ::/0
!
vrf vrf_eVPNZ1
 vni 102030
exit-vrf
!
interface enp1s0d1
 ip router openfabric 1
exit
!
interface lo
 ip address 10.255.0.203/32
 ip router openfabric 1
 openfabric passive
exit
!
router bgp 65000
 bgp router-id 10.255.0.203
 no bgp enforce-first-as
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 bgp disable-ebgp-connected-route-check
 coalesce-time 1000
 no bgp graceful-restart notification
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 2
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor VTEP ebgp-multihop 10
 neighbor VTEP update-source lo
 neighbor 10.255.0.10 peer-group BGP
 neighbor 10.255.0.201 peer-group VTEP
 neighbor 10.255.0.202 peer-group VTEP
 !
 address-family ipv4 unicast
  network 10.255.0.203/32
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_eVPNZ1
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_eVPNZ1
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
  autort as 65040
 exit-address-family
exit
!
router bgp 65000 vrf vrf_eVPNZ1
 bgp router-id 10.255.0.203
 no bgp enforce-first-as
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
  route-target import 65040:102030
  route-target export 65040:102030
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match ip address prefix-list only_default
exit
!
route-map MAP_VTEP_IN deny 2
 match ipv6 address prefix-list only_default_v6
exit
!
route-map MAP_VTEP_IN permit 3
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
route-map correct_src permit 1
 match ip address prefix-list loopbacks_ips
 set src 10.255.0.203
exit
!
ip protocol bgp route-map correct_src
!
router openfabric 1
 net 49.0001.3333.3333.3333.00
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180
exit
!
end

and here is the output of "show bgp evpn summary":

Code: 
# show bgp evpn summary
BGP router identifier 10.255.0.203, local AS number 65000 VRF default vrf-id 0
BGP table version 0
RIB entries 5, using 640 bytes of memory
Peers 2, using 47 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor            V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
pve01(10.255.0.201) 4      65040         6         3        0    0    0    never         Idle        0 FRRouting/10.2.2
pve02(10.255.0.202) 4      65040         9         4        0    0    0    never         Idle        0 FRRouting/10.2.2

Total number of neighbors 2


And here is the output of the same command when I remove the BGP controller:

Code: 
# show bgp evpn summary
BGP router identifier 10.255.0.203, local AS number 65040 VRF default vrf-id 0
BGP table version 0
RIB entries 13, using 1664 bytes of memory
Peers 2, using 47 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor            V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
pve01(10.255.0.201) 4      65040        20        17        5    0    0 00:00:21            5        3 FRRouting/10.2.2
pve02(10.255.0.202) 4      65040        16        17        5    0    0 00:00:21            1        3 FRRouting/10.2.2

Total number of neighbors 2


And here is the FRR config without the BGP controller:

Code: 
# show running-config
Building configuration...

Current configuration:
!
frr version 10.2.2
frr defaults datacenter
hostname pve03
log syslog informational
service integrated-vtysh-config
!
ip prefix-list only_default seq 1 permit 0.0.0.0/0
!
ipv6 prefix-list only_default_v6 seq 1 permit ::/0
!
vrf vrf_eVPNZ1
 vni 102030
exit-vrf
!
interface enp1s0d1
 ip router openfabric 1
exit
!
interface lo
 ip address 10.255.0.203/32
 ip router openfabric 1
 openfabric passive
exit
!
router bgp 65040
 bgp router-id 10.255.0.203
 no bgp enforce-first-as
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 coalesce-time 1000
 no bgp graceful-restart notification
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65040
 neighbor VTEP bfd
 neighbor 10.255.0.201 peer-group VTEP
 neighbor 10.255.0.202 peer-group VTEP
 !
 address-family ipv4 unicast
  import vrf vrf_eVPNZ1
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_eVPNZ1
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65040 vrf vrf_eVPNZ1
 bgp router-id 10.255.0.203
 no bgp enforce-first-as
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match ip address prefix-list only_default
exit
!
route-map MAP_VTEP_IN deny 2
 match ipv6 address prefix-list only_default_v6
exit
!
route-map MAP_VTEP_IN permit 3
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
router openfabric 1
 net 49.0001.3333.3333.3333.00
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180
exit
!
end
 
Can you post the output of
Code: 
show bfd peers

when EVPN is not working
 
Last edited:
Yes, here it is:


Code: 
# show bfd peers
BFD Peers:
    peer 10.255.0.201 multihop local-address 10.255.0.203 vrf default
        ID: 426939240
        Remote ID: 0
        Active mode
        Minimum TTL: 246
        Status: down
        Downtime: 35 second(s)
        Diagnostics: neighbor signaled session down
        Remote diagnostics: neighbor signaled session down
        Peer Type: dynamic
        RTT min/avg/max: 0/0/0 usec
        Local timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo receive interval: 50ms
            Echo transmission interval: disabled
        Remote timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo receive interval: 50ms

    peer 10.255.0.202 multihop local-address 10.255.0.203 vrf default
        ID: 1075973845
        Remote ID: 0
        Active mode
        Minimum TTL: 246
        Status: down
        Downtime: 25 second(s)
        Diagnostics: neighbor signaled session down
        Remote diagnostics: neighbor signaled session down
        Peer Type: dynamic
        RTT min/avg/max: 0/0/0 usec
        Local timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo receive interval: 50ms
            Echo transmission interval: disabled
        Remote timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo receive interval: 50ms
 
Seems like BFD is the issue here, we'll be deploying a bacported fix for FRR very soon (10.2.3).
 
Ok, thanks a lot! Do I mark as "resolved" right now, or do I wait and report back to you after I test v10.2.3 ?
 
geforce said:
Ok, thanks a lot! Do I mark as "resolved" right now, or do I wait and report back to you after I test v10.2.3 ?
Click to expand...
FRR 10.2.3 should now be on the no-subscription repository, so if you could give that version a spin - that would be great!
 
shanreich said:
FRR 10.2.3 should now be on the no-subscription repository, so if you could give that version a spin - that would be great!
Click to expand...
Hi,

I ended up upgrading FRR (which didn't seem to fix -- but I didn't test the finding below regarding the ASN), and then upgrading the whole pve cluster to v9 because I needed to test fabrics for my use case. PVE V9 reports with FRR v10.3.1.

As for the (e)BGP Controller + EVPN controller combo, good news and bad news:

  • The bad news is that the only way to make the (e)bgp controller work simultaniously as the EVPN controller is to use the same ASN for both controllers. If you use a different ASN, the EVPN controller breaks for that node, in the same manner I reported ealier in this thread.

  • The good news is that, while it's somewhat of a "limitation", it now work A1 when using the same ASN (as I recall, it previously didn't). So this is very good!

The only other limitation I found, which is somewhat expected I guess, is that for some reason pve nodes can't route trafic for VM/CTs that are running on another pve node.

For example: if I have 3 nodes (PVE1, PVE2, PVE3) which are iBGP peers for a given EVPN controller and VRF/Zone, and I then create an eBGP controller on PVE3 to leak routes to my router, then only the VM/CTs that are running on PVE3 will respond. Using the "exit nodes local routing" option didn't help; a given node still can't reach VM/CTs running on another node.

So this means that you need to either 1) make your router an EVPN ibgp peer, or 2) set the "designated route-leaking node" as the default exit node for the EVPN.

So, I'd say that there are still maybe "rough edges" (or unexpected behaviours) depending on the use case, but it's still a damn fine set of features.
Thanks a lot! I'll mark the thread as resolved.
 
You must log in or register to reply here.
Top Bottom