help: eBGP is not working

[imoniker]

Dear Spirit,

I'm setting up a 2 nodes test environment, with one VM on each node (IP: 10.41.10.170, 10.41.10.171), the two VM cannot ping each other.
Is there something wrong with my config?
Node 1:

root@pve5h1:~# cat /etc/pve/sdn/*.cfg evpn: c01 asn 65170 peers 192.168.223.170 bgp: bgppve5h1 asn 65172 node pve5h1 peers 192.168.223.171 bgp-multipath-as-path-relax 0 ebgp 1 ebgp-multihop 10 loopback dummy0 subnet: zone1-10.41.10.1-24 vnet vnet1 gateway 10.41.10.1 vnet: vnet1 zone zone1 tag 1170 evpn: zone1 controller c01 vrf-vxlan 10170 advertise-subnets 1 disable-arp-nd-suppression 1 ipam pve mac 0E:DA:6F:D4:02:48

Node 2:

root@pve5h2:~# cat /etc/pve/sdn/*.cfg
evpn: c01
        asn 65170
        peers 192.168.223.171

bgp: bgppve5h2
        asn 65173
        node pve5h2
        peers 192.168.223.170
        bgp-multipath-as-path-relax 0
        ebgp 1
        ebgp-multihop 10
        loopback dummy0

subnet: zone1-10.41.10.1-24
        vnet vnet1
        gateway 10.41.10.1

vnet: vnet1
        zone zone1
        tag 1170

evpn: zone1
        controller c01
        vrf-vxlan 10170
        advertise-subnets 1
        disable-arp-nd-suppression 1
        ipam pve
        mac E6:A1:3E:49:F3:AC

-------------------------------------Vtysh on Node 2, same result on Node1-------------------------------------

pve5h2# sh bgp sum IPv4 Unicast Summary (VRF default): BGP router identifier 192.168.223.171, local AS number 65173 vrf-id 0 BGP table version 2 RIB entries 1, using 192 bytes of memory Peers 1, using 725 KiB of memory Peer groups 2, using 128 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc pve5h1(192.168.223.170) 4 65172 543 543 0 0 0 00:15:21 0 0 N/A Total number of neighbors 1 pve5h2# sh bgp l2vpn evpn BGP table version is 9, local router ID is 192.168.223.171 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id] EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 192.168.223.171:2 *> [5]:[0]:[24]:[10.41.10.0] 192.168.223.171(pve5h2) 0 32768 ? ET:8 RT:65170:10170 Rmac:16:4e:12:51:73:f5 Route Distinguisher: 192.168.223.171:3 *> [2]:[0]:[48]:[52:b1:27:4a:ab:6b] 192.168.223.171(pve5h2) 32768 i ET:8 RT:65170:1170 *> [2]:[0]:[48]:[52:b1:27:4a:ab:6b]:[32]:[10.40.10.171] 192.168.223.171(pve5h2) 32768 i ET:8 RT:65170:1170 RT:65170:10170 Rmac:16:4e:12:51:73:f5 *> [2]:[0]:[48]:[52:b1:27:4a:ab:6b]:[32]:[10.41.10.171] 192.168.223.171(pve5h2) 32768 i ET:8 RT:65170:1170 RT:65170:10170 Rmac:16:4e:12:51:73:f5 *> [3]:[0]:[32]:[192.168.223.171] 192.168.223.171(pve5h2) 32768 i ET:8 RT:65170:1170 Displayed 5 out of 5 total prefixes

 

[imoniker]

Same subnet in two AS caused the problem：
10.41.10.170 cannot ping 10.41.10.171
10.41.10.170 can ping 10.42.10.171 if subnet 10.41.10.1/24 was deleted on host 2.
Couldn't we use the same subnet with eBGP?

 

[imoniker]

it seems the address-family l2vpn evpn is missing in "show bgp nei 192.168.223.170"

pve5h2# show bgp nei 192.168.223.170
BGP neighbor is 192.168.223.170, remote AS 65172, local AS 65173, external link
  Local Role: undefined
  Remote Role: undefined
Hostname: pve5h1
 Member of peer-group BGP for session parameters
  BGP version 4, remote router ID 192.168.223.170, local router ID 192.168.223.171
  BGP state = Established, up for 00:11:33
  Last read 00:00:03, Last write 00:00:03
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised and received
    Long-lived Graceful Restart: advertised and received
      Address families by peer:
    Route refresh: advertised and received(old & new)
    Enhanced Route Refresh: advertised and received
    Address Family IPv4 Unicast: advertised and received
    Hostname Capability: advertised (name: pve5h2,domain name: n/a) received (name: pve5h1,domain name: n/a)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received: IPv4 Unicast
    Local GR Mode: Helper*

    Remote GR Mode: Helper

    R bit: True
    N bit: True
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
    IPv4 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                 48          6
    Notifications:          5          0
    Updates:               12         18
    Keepalives:          1724       1722
    Route Refresh:          0          0
    Capability:             0          0
    Total:               1789       1746
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast
  BGP peer-group member
  Update group 6, subgroup 6
  Packet Queue length 0
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  1 accepted prefixes

  Connections established 6; dropped 5
  Last reset 00:11:37,  No AFI/SAFI activated for peer
  External BGP neighbor may be up to 10 hops away.
Local host: 192.168.223.171, Local port: 179
Foreign host: 192.168.223.170, Foreign port: 37710
Nexthop: 192.168.223.171
Nexthop global: fe80::20c:29ff:fe57:d449
Nexthop local: fe80::20c:29ff:fe57:d449
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Estimated round trip time: 0 ms
Read thread: on  Write thread: on  FD used: 27

  BFD: Type: multi hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Up, Last update: 0:00:11:32

and here is my generated frr.conf:

root@pve5h2:~# cat /etc/frr/frr.conf
frr version 8.5.1
frr defaults datacenter
hostname pve5h2
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_zone1
 vni 10170
exit-vrf
!
router bgp 65173
 bgp router-id 192.168.223.171
 no bgp default ipv4-unicast
 coalesce-time 1000
 bgp disable-ebgp-connected-route-check
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 10
 neighbor 192.168.223.170 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor VTEP ebgp-multihop 10
 neighbor VTEP update-source dummy0
 neighbor 192.168.223.170 peer-group VTEP
 !
 address-family ipv4 unicast
  network 192.168.223.171/32
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_zone1
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_zone1
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65170
 exit-address-family
exit
!
router bgp 65173 vrf vrf_zone1
 bgp router-id 192.168.223.171
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65170:10170
  route-target export 65170:10170
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
ip prefix-list loopbacks_ips seq 10 permit 0.0.0.0/0 le 32
!
route-map MAP_VTEP_IN deny 1
 match evpn vni 10170
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
route-map correct_src permit 1
 match ip address prefix-list loopbacks_ips
 set src 192.168.223.171
exit
!
ip protocol bgp route-map correct_src
!
line vty

 

[spirit]

what is the ip of pve5h1 && pve5h2 ?

is it : pve5h1 : 192.168.223.170 ?

and

pve5h2 : 192.168.223.171 ?



I'm a bit lot, because I see In your config:

pve5h1 ----> evpn ---> peer 192.168.223.170
------> bgp -----> peer 192.168.223.171

pve5h2 ---> evpn---> peer 192.168.223.171
----> bgp ----> peer 192.168.223.170


I don't see how it could work, as evpn for evpn, the peer it's configured with the node itself.

Any special reason to use ebgp here ?

you could simply use

evpn: c01
        asn 65170
        peers 192.168.223.170,192.168.223.171

on both nodes. (the itself ip will be autofiltered by the code)

without extra bgp controller

 

[imoniker]

what is the ip of pve5h1 && pve5h2 ?

is it : pve5h1 : 192.168.223.170 ?

and

pve5h2 : 192.168.223.171 ?



I'm a bit lot, because I see In your config:

pve5h1 ----> evpn ---> peer 192.168.223.170
------> bgp -----> peer 192.168.223.171

pve5h2 ---> evpn---> peer 192.168.223.171
----> bgp ----> peer 192.168.223.170


I don't see how it could work, as evpn for evpn, the peer it's configured with the node itself.

Any special reason to use ebgp here ?

you could simply use

evpn: c01
        asn 65170
        peers 192.168.223.170,192.168.223.171

on both nodes. (the itself ip will be autofiltered by the code)

without extra bgp controller

I did this just for testing purposes.

I already did the test without BGP controller and it worked great with full-mesh peers or route reflector.

Obviously my previous settings were not correct. I'm quite new to SDN and BGP etc.

What I want to test is something like this:

"each hypervisor establishes an eBGP session with TOR router"

I guess I need a Debian to emulate a TOR router and two PVE hypervisors.

Is it possible to do this? If you have a tutorial with PVE and FRR config files that will be very helpful

 

[spirit]

yes, sure . I'm doing at work .

if your tor router can't do evpn, but only bgp, you can do something like

- for evpn : do a full mesh peers betwen proxmox nodes or use a route reflectors
- add a bgp controller for each node , and add your tor(s) router(s) ip as peer.

(Then your tor(s) router(s) need to peers themself with the spine for examples)

This is exactly my setup at work. (for evpn, I'm using arista routers as route-reflectors and also using them as evpn exit-node)