SDN bgp-evpn not working

A

antubis

Member
Apr 20, 2012
14
0
21
In a (GNS3 based) test setup I'm currently trying to implement a BGP-EVPN setup.

As we are using Dell OS10 switches I built a setup based on the example from...
https://www.dell.com/support/manual...bf9452-d9a4-4822-a715-d63069e40b8b&lang=en-us
...without the overlay VRF- and layer3 routing part (means only BGP underlay routing and EVPN for distributing the VXLAN vtep ips).

I want now to integrate a PVE host as an equal single-host leaf into this setup and somehow can't get it to work. BGP underlay routing works but I can't get it to exchange the vtep ips; both sides (dell switch leafs and pve leaf) are ignoring each other.

Any advice is appreciated.
 
Hi, could you share your config on both dell && on proxmox /etc/pve/sdn/*.cfg ?

also, on 1 proxmox node, can you send the result of

Code: 
vtysh -c " sh bgp summary"
vtysh -c "sh bgp l2vpn evpn"


abou layer3 routing, the proxmox code create by default a vrf by zone (with anycast gateway && symetric routing), but it should be a problem if on dell you use asymetric or even no routing, with simple layer2 vxlan.

on dell side, from doc, maybe:

Configure the EVPN instance, RD, and RT using auto-EVI mode:

OS10(config)# evpn
OS10(config-evpn)# auto-evi
OS10(config-evpn)# exit
Click to expand...

need some manual to tuning for RT

proxmox code use AS number:vnet tag for RT
(for example, with AS 65000 and vnet with tag 1000 , RT= 65000:1000)
 
Dell leaf switch:
Code: 
sw02-r12# show running-configuration nve 
!
nve
 source-interface loopback0

sw02-r12# show running-configuration bgp
!
router bgp 65100
 router-id 172.18.0.1
 !
 address-family ipv4 unicast
  redistribute connected
 !
 neighbor 172.18.1.1
  remote-as 65101
  no shutdown
  !
  address-family ipv4 unicast
   allowas-in 1
 !
 neighbor 172.18.2.1
  remote-as 65101
  no shutdown
  !
  address-family ipv4 unicast
   allowas-in 1
 !
 neighbor 172.201.0.1
  ebgp-multihop 4
  remote-as 65101
  send-community extended
  update-source loopback1
  no shutdown
  !
  address-family ipv4 unicast
   no activate
  !
  address-family l2vpn evpn
   activate
   allowas-in 1
 !
 neighbor 172.202.0.1
  ebgp-multihop 4
  remote-as 65101
  send-community extended
  update-source loopback1
  no shutdown
  !
  address-family ipv4 unicast
   no activate
  !
  address-family l2vpn evpn
   activate
   allowas-in 1


sw02-r12# show virtual-network
Codes: DP - MAC-learn Dataplane, CP - MAC-learn Controlplane, UUD - Unknown-Unicast-Drop
Un-tagged VLAN: 4001
Virtual Network: 10000
   VLTi-VLAN: 100
   Members:
      Untagged: port-channel10
      VLAN 100: port-channel1000
   VxLAN Virtual Network Identifier: 10000
      Source Interface: loopback0(192.168.2.1)
      Remote-VTEPs (flood-list): 192.168.1.1(CP),192.168.3.1(CP),192.168.111.1(CP)

 
sw02-r12# show evpn evi

EVI : 10000, State : up
  Bridge-Domain       : Virtual-Network 10000, VNI 10000
  Route-Distinguisher : 1:192.168.2.1:10000(auto)
  Route-Targets       : 0:65100:268445456(auto) both
  Inclusive Multicast : 192.168.1.1, 192.168.3.1, 192.168.111.1
  IRB                 : Disabled


Dell spine switch:
Code: 
spine1# show running-configuration bgp 
!
router bgp 65101
 router-id 172.201.0.1
 !
 address-family ipv4 unicast
  redistribute connected
 !
 neighbor 172.18.1.0
  remote-as 65100
  no shutdown
  !
  address-family ipv4 unicast
   no sender-side-loop-detection
 !
 neighbor 172.18.0.1
  ebgp-multihop 4
  remote-as 65100
  send-community extended
  update-source loopback1
  no shutdown
  !
  address-family ipv4 unicast
   no activate
  !
  address-family l2vpn evpn
   activate
   no sender-side-loop-detection
 !
 neighbor 172.22.1.0
  remote-as 65100
  no shutdown
  !
  address-family ipv4 unicast
   no sender-side-loop-detection
 !
 neighbor 172.22.0.1
  ebgp-multihop 4
  remote-as 65100
  send-community extended
  update-source loopback1
  no shutdown
  !
  address-family ipv4 unicast
   no activate
  !
  address-family l2vpn evpn
   activate
   no sender-side-loop-detection

PVE sdn config:
Code: 
# controller.cfg
bgp: bgppve-testhost1
        asn 65100
        node pve-testhost1
        peers 172.22.1.1,172.22.2.1
        ebgp 1
        ebgp-multihop 4
        loopback dummy1

evpn: evpn1
        asn 65101
        peers 172.201.0.1,192.202.0.2

# zones.cfg
evpn: evpn1
        controller evpn1
        vrf-vxlan 100
        ipam pve
        mac AA:BB:CC:DD:EE:FF
        
# vnets.cfg
vnet: vx10000
        zone evpn1
        tag 10000
        vlanaware 1


Code: 
# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 172.22.0.1, local AS number 65100 vrf-id 0
BGP table version 17
RIB entries 65, using 12 KiB of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
172.22.1.1      4      65101     38947     33300        0    0    0 1d03h44m            8       17 N/A
172.22.2.1      4      65101     38937     33300        0    0    0 1d03h44m            8       17 N/A

Total number of neighbors 2

L2VPN EVPN Summary:
BGP router identifier 172.22.0.1, local AS number 65100 vrf-id 0
BGP table version 0
RIB entries 15, using 2760 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
172.201.0.1     4      65101       830       660        0    0    0 00:32:26            0        1 N/A
172.202.0.1     4      65101       832       660        0    0    0 00:32:26            0        1 N/A


# show bgp l2vpn evpn
BGP table version is 4, local router ID is 172.22.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 172.22.0.1:4
*> [3]:[0]:[32]:[172.22.0.1]
                    172.22.0.1(pve-testhost1)
                                                       32768 i
                    ET:8 RT:65101:10000

Displayed 1 out of 1 total prefixes

IPs:
172.18.1.0/31 <-> 172.18.1.1/31 leaf switch / spine1
172.18.2.0/31 <-> 172.18.2.1/31 leaf switch / spine2
172.18.0.1/32 leaf loopback1 (bgp router id)
192.168.2.1/32 leap loopback0 (nve / evpn source ip)

172.22.1.0/31 <-> 172.22.1.1/31 pve-host / spine1
172.22.2.0/31 <-> 172.22.2.1/31 pve-host / spine12
172.22.0.1/32 <-> pve-host dummy1 (bgp router id)

172.201.0.1/32 spine1 loopback1 (bgp router id)
172.201.0.2/32 spine2 loopback1 (bgp router id)


I just found one problem... I previously configured in the evpn controller the leaf AS 65100 instead of the spine AS 65101. Now the show bgp neighbors 172.201.0.1 at least shows me some Update Group X membership, beforehand it just was Not part of any update group.

To get around removing the auto-evi option on the Dell side (I don't want to define an RT for all planned VNIs manually) I tried to set route-target both 65100:268445456 in the pve host vtysh, but it didn't help in any possible way to get it to work.
 
I have added an option "rt-import" for route target import (only import, no import/export = both) in zones.cfg recently ( libpve-network-perl 0.7.0), but it's not yet available in gui.

in /etc/pve/sdn/zones.cfg

example:

Code: 
evpn: yourzone
        controller evpnctl
        vrf-vxlan 10000
        rt-import 65000:100,65001:200,....


then use "apply sdn button" in gui to push config



I'll look at your setup today

BTW, you could also use another sdn for your proxmox node. Personaly, at work, you use ebgp with 1 differents asn for each proxmox node)/ This should avoid to have Allowas-in in dell config.
For RT, proxmox will always use asn defined in evpn controller. (even if differents nodes have specific bgp controllers with differents asn)
 
Last edited:
Code: 
# show bgp l2vpn evpn vni

Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
Number of L2 VNIs: 1
Number of L3 VNIs: 1
Flags: * - Kernel
  VNI        Type RD                    Import RT                 Export RT                 Tenant VRF                          
* 10000      L2   172.22.0.1:4          65101:10000               65101:10000              vrf_evpn1                          
* 100        L3   172.22.0.1:2          65100:268445456, ...      65101:100                vrf_evpn1

Tried the rt-import option with the Dell exported RTs. For me it looks like it's on the wrong VNI. In my testsetup the 100 is the proxmox-mandatory L3 VRF VNI, but I need the import (and the export as well) on the VNI 10000 L2 vnet.


BTW, you could also use another sdn for your proxmox node. Personaly, at work, you use ebgp with 1 differents asn for each proxmox node)/ This should avoid to have Allowas-in in dell config.
Click to expand...

Yeah I also thought about using the Dell multi-as-config example but with this you can't use the auto-evi feature.
When we migrate the whole setup to production there will be ~30 leafs to be configured, as each leaf will be a VLT pair it's 60 switches. So I'd really appreciate if I don't have to configure RTs on each device manually.
 
Last edited:
if you look in /etc/frr/frr.conf,
you will see something like

Code: 
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65101
 exit-address-family

this auto import|export RT for all l2vni, like : <evpn controller asn>:<vnet tag>.

So, I think the more simple should be to match the RT of your dell switches.

so, in evpn controller: use 65100 as ASN (used for the RT) . (when you define an extra bgp controller, the asn defined in the evpn controller, is really only used for the RT)
 
After a lot of try and error finally it works!

Here is my frr.conf, based on the automatically generated SDN configuration:
Code: 
frr version 8.0.1
frr defaults datacenter
hostname pve-testhost1
log syslog informational
service integrated-vtysh-config
!
vrf vrf_evpn1
 vni 100
 exit-vrf
!
router bgp 65100
 bgp router-id 172.22.0.1
 no bgp default ipv4-unicast
 bgp disable-ebgp-connected-route-check
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor BGP capability extended-nexthop
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor VTEP ebgp-multihop 10
 neighbor VTEP update-source dummy1
 neighbor VTEP capability extended-nexthop
 neighbor 172.22.1.1 peer-group BGP
 neighbor 172.22.2.1 peer-group BGP
 neighbor 172.201.0.1 peer-group VTEP
 neighbor 172.202.0.1 peer-group VTEP
 !
 address-family ipv4 unicast
  network 172.22.0.1/32
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  neighbor BGP allowas-in 1
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP allowas-in 1
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
  vni 10000
   route-target import 65100:268445456
   route-target export 65100:268445456
  exit-vni
 exit-address-family
!
router bgp 65100 vrf vrf_evpn1
 bgp router-id 172.22.0.1
 !
 address-family l2vpn evpn
  route-target import 65101:100
  route-target export 65101:100
 exit-address-family
!
ip prefix-list loopbacks_ips seq 10 permit 0.0.0.0/0 le 32
!
route-map MAP_VTEP_OUT permit 1
!
route-map correct_src permit 1
 match ip address prefix-list loopbacks_ips
 set src 172.22.0.1
!
ip protocol bgp route-map correct_src
!
line vty

After configuring the route-target import / route-target export *within* the VNI definition (not just in the address-family section) I finally saw the PVE bgp router id in my Dell switches, but still no remote VTEPs in the pve host. For these to work I added the allowas-in 1 as well as the capability extended-nexthop options for both peer-groups (BGP and VTEP).

Now in the show vni evpn detail I finally can see the Dell switches as remote VTEPs and the ping in the container using the vnet device is working.

So it seems I have to choose if I want to define the VNIs with its RTs manually either on PVE side or on the Dell switches. As the PVE hosts will be fewer I think I'll go with these.
Is there any chance that doing these FRR changes I did will be included in the GUI somehow in the future?
At the moment clicking on Apply In the SDN config kills these adjustments - at least an option to make them permanent should be available, I think.
 
I think it's a very specific setup, and currently I don't plan to add specific plugin option on "vnet" part. (vnets are common and shared between differents differents zones types).
(as here, it would need route target import on vnets)


But, they are an hidden trick, to add your custom frr config:

simply add your config in:

/etc/frr/frr.conf.local

The sdn plugin will used it on apply button to generate the /etc/fr/frr.conf.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back