Proxmox EVPN iBGP to Arista SW

kemeris

Member
Nov 23, 2021
25
0
6
45
Hello everyone,

I am trying to connect my 3 node Proxmox cluster to Arista 7060CX switch via iBGP so Arista could act as "Exit node" for Proxmox SDN Zones. I am new to Arista.
I have managed to establish iBGP session with Arista, map evpn routes to corresponding VRF's and VLANS to mimic network topology on Proxmox side. My problem is I can't ping VM in certain SDN zone from Arista, ICMP does reach Proxmox node, but never reach VM (10.0.20.8), even when I can ping VM from node itself. Any ideas what could be wrong?


Proxmox node: /sdn/zones.cfg:
Code:
evpn: z10001
        controller evpn1
        vrf-vxlan 10001
        advertise-subnets 1
        exitnodes proxmox1-3,proxmox1-1,proxmox1-4
        ipam pve
        mac BC:24:11:35:D0:42
        mtu 1500

evpn: z10002
        controller evpn1
        vrf-vxlan 10002
        advertise-subnets 1
        exitnodes proxmox1-1,proxmox1-3,proxmox1-4
        ipam pve
        mac BC:24:11:A8:42:37
        mtu 1500

evpn: z10007
        controller evpn1
        vrf-vxlan 10007
        advertise-subnets 1
        exitnodes proxmox1-4,proxmox1-3,proxmox1-1
        ipam pve
        mac BC:24:11:E8:E4:99
        mtu 1500

evpn: z10009
        controller evpn1
        vrf-vxlan 10009
        advertise-subnets 1
        exitnodes proxmox1-1,proxmox1-3,proxmox1-4
        ipam pve
        mac BC:24:11:AF:D1:11
        mtu 1500

evpn: z10015
        controller evpn1
        vrf-vxlan 10015
        advertise-subnets 1
        exitnodes proxmox1-3,proxmox1-1,proxmox1-4
        ipam pve
        mac BC:24:11:09:D9:27
        mtu 1500

evpn: z10013
        controller evpn1
        vrf-vxlan 100013
        advertise-subnets 1
        disable-arp-nd-suppression 0
        exitnodes proxmox1-4,proxmox1-3,proxmox1-1
        exitnodes-local-routing 0
        ipam pve
        mac BC:24:11:FA:77:CC
        mtu 1500


Proxmox node: /sdn/vnets.conf
Code:
vnet: v100001
        zone z10001
        tag 200001

vnet: v100007
        zone z10007
        tag 200007

vnet: v100009
        zone z10009
        tag 200009

vnet: v100015
        zone z10015
        tag 200015

vnet: v100002
        zone z10002
        tag 200002

vnet: v100013
        zone z10013
        tag 200013


Proxmox node: /sdn/subnets.conf
Code:
subnet: z10001-10.0.20.0-24
        vnet v100001
        gateway 10.0.20.1
        snat 1

subnet: z10007-10.0.22.0-24
        vnet v100007
        gateway 10.0.22.1
        snat 1

subnet: z10009-10.0.23.0-24
        vnet v100009
        gateway 10.0.23.1
        snat 1

subnet: z10015-10.0.24.0-24
        vnet v100015
        gateway 10.0.24.1
        snat 1

subnet: z10001-2001:1ab9:f002:2::-118
        vnet v100001
        gateway 2001:1ab9:f002:2::1

subnet: z10002-10.0.21.0-24
        vnet v100002
        gateway 10.0.21.1
        snat 1

subnet: z10013-10.0.24.0-24
        vnet v100013
        gateway 10.0.24.1
        snat 1
 
Last edited:
Arista: received routes from Proxmox:
Code:
arista7060cx#show bgp neighbors 10.0.4.1 evpn received-routes detail
BGP routing table information for VRF default
Router identifier 192.168.10.1, local AS number 65000
BGP routing table entry for mac-ip 0aa9.2c3c.b7a0, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 0aa9.2c3c.b7a0 10.0.20.8, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10001 Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:ce:ec:f4:6c:d0:d1
      VNI: 200001 L3 VNI: 10001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 0aa9.2c3c.b7a0 fe80::8a9:2cff:fe3c:b7a0, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 1afb.635c.774f, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 1afb.635c.774f fe80::18fb:63ff:fe5c:774f, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 4a70.56de.7973, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 4a70.56de.7973 fe80::4870:56ff:fede:7973, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 5205.b645.ad4d, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip bc24.1126.9cbb, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip bc24.1126.9cbb 10.0.20.42, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10001 Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:ce:ec:f4:6c:d0:d1
      VNI: 200001 L3 VNI: 10001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip bc24.1128.99d8, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip bc24.1128.99d8 2001:1ab9:f002:2::6e, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10001 Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:ce:ec:f4:6c:d0:d1
      VNI: 200001 L3 VNI: 10001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip bc24.1128.99d8 fe80::be24:11ff:fe28:99d8, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip e6cb.1735.c2d2, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip e6cb.1735.c2d2 2001:1ab9:f002:2::73, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10001 Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:ce:ec:f4:6c:d0:d1
      VNI: 200001 L3 VNI: 10001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip e6cb.1735.c2d2 fe80::e4cb:17ff:fe35:c2d2, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for imet 10.0.4.1, Route Distinguisher: 10.0.4.1:8
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200001 TunnelEncap:tunnelTypeVxlan
      VNI: 200001
      PMSI Tunnel: Ingress Replication, MPLS Label: 200001, Leaf Information Required: false, Tunnel ID: 10.0.4.1
BGP routing table entry for imet 10.0.4.1, Route Distinguisher: 10.0.4.1:10
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200002 TunnelEncap:tunnelTypeVxlan
      VNI: 200002
      PMSI Tunnel: Ingress Replication, MPLS Label: 200002, Leaf Information Required: false, Tunnel ID: 10.0.4.1
BGP routing table entry for imet 10.0.4.1, Route Distinguisher: 10.0.4.1:12
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200007 TunnelEncap:tunnelTypeVxlan
      VNI: 200007
      PMSI Tunnel: Ingress Replication, MPLS Label: 200007, Leaf Information Required: false, Tunnel ID: 10.0.4.1
BGP routing table entry for imet 10.0.4.1, Route Distinguisher: 10.0.4.1:14
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200009 TunnelEncap:tunnelTypeVxlan
      VNI: 200009
      PMSI Tunnel: Ingress Replication, MPLS Label: 200009, Leaf Information Required: false, Tunnel ID: 10.0.4.1
BGP routing table entry for imet 10.0.4.1, Route Distinguisher: 10.0.4.1:16
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200013 TunnelEncap:tunnelTypeVxlan
      VNI: 200013
      PMSI Tunnel: Ingress Replication, MPLS Label: 200013, Leaf Information Required: false, Tunnel ID: 10.0.4.1
BGP routing table entry for imet 10.0.4.1, Route Distinguisher: 10.0.4.1:18
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:200015 TunnelEncap:tunnelTypeVxlan
      VNI: 200015
      PMSI Tunnel: Ingress Replication, MPLS Label: 200015, Leaf Information Required: false, Tunnel ID: 10.0.4.1
BGP routing table entry for ip-prefix 0.0.0.0/0, Route Distinguisher: 10.0.4.1:2
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10001 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:ce:ec:f4:6c:d0:d1
      VNI: 0
BGP routing table entry for ip-prefix 0.0.0.0/0, Route Distinguisher: 10.0.4.1:3
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10002 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:4a:a3:1f:ac:af:e0
      VNI: 0
BGP routing table entry for ip-prefix 0.0.0.0/0, Route Distinguisher: 10.0.4.1:4
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10007 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:62:45:6e:b5:ba:84
      VNI: 0
BGP routing table entry for ip-prefix 0.0.0.0/0, Route Distinguisher: 10.0.4.1:5
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10009 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:2e:77:bc:ca:4a:45
      VNI: 0
BGP routing table entry for ip-prefix 0.0.0.0/0, Route Distinguisher: 10.0.4.1:6
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:100013 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:66:fb:b5:de:84:13
      VNI: 0
BGP routing table entry for ip-prefix 0.0.0.0/0, Route Distinguisher: 10.0.4.1:7
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10015 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:ba:cf:9b:87:57:7d
      VNI: 0
BGP routing table entry for ip-prefix ::/0, Route Distinguisher: 10.0.4.1:2
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10001 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:ce:ec:f4:6c:d0:d1
      VNI: 0
BGP routing table entry for ip-prefix ::/0, Route Distinguisher: 10.0.4.1:3
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10002 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:4a:a3:1f:ac:af:e0
      VNI: 0
BGP routing table entry for ip-prefix ::/0, Route Distinguisher: 10.0.4.1:4
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10007 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:62:45:6e:b5:ba:84
      VNI: 0
BGP routing table entry for ip-prefix ::/0, Route Distinguisher: 10.0.4.1:5
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10009 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:2e:77:bc:ca:4a:45
      VNI: 0
BGP routing table entry for ip-prefix ::/0, Route Distinguisher: 10.0.4.1:6
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:100013 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:66:fb:b5:de:84:13
      VNI: 0
BGP routing table entry for ip-prefix ::/0, Route Distinguisher: 10.0.4.1:7
 Paths: 1 available
  Local
    10.0.4.1 from 10.0.4.1 (10.0.4.1)
      Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
      Extended Community: Route-Target-AS:65000:10015 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:ba:cf:9b:87:57:7d
      VNI: 0
 
Arista: VRF routing table
Code:
arista7060cx#show ip route vrf all

VRF: default
Source Codes:
       C - connected, S - static, K - kernel,
       O - OSPF, O IA - OSPF inter area, O E1 - OSPF external type 1,
       O E2 - OSPF external type 2, O N1 - OSPF NSSA external type 1,
       O N2 - OSPF NSSA external type2, O3 - OSPFv3,
       O3 IA - OSPFv3 inter area, O3 E1 - OSPFv3 external type 1,
       O3 E2 - OSPFv3 external type 2,
       O3 N1 - OSPFv3 NSSA external type 1,
       O3 N2 - OSPFv3 NSSA external type2, B - Other BGP Routes,
       B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
       I L2 - IS-IS level 2, A B - BGP Aggregate,
       A O - OSPF Summary, NG - Nexthop Group Static Route,
       V - VXLAN Control Service, M - Martian,
       DH - DHCP client installed default route,
       DP - Dynamic Policy Route, L - VRF Leaked,
       G  - gRIBI, RC - Route Cache Route,
       CL - CBF Leaked Route

Gateway of last resort:
 S        0.0.0.0/0 [1/0]
           via 5.133.66.1, Ethernet34

 C        5.133.66.0/24
           directly connected, Ethernet34
 C        10.0.4.0/22
           directly connected, Vlan3
 C        10.0.8.0/22
           directly connected, Vlan4
 C        10.0.12.0/22
           directly connected, Vlan5
 C        192.168.10.1/32
           directly connected, Loopback0


VRF: 10001
Source Codes:
       C - connected, S - static, K - kernel,
       O - OSPF, O IA - OSPF inter area, O E1 - OSPF external type 1,
       O E2 - OSPF external type 2, O N1 - OSPF NSSA external type 1,
       O N2 - OSPF NSSA external type2, O3 - OSPFv3,
       O3 IA - OSPFv3 inter area, O3 E1 - OSPFv3 external type 1,
       O3 E2 - OSPFv3 external type 2,
       O3 N1 - OSPFv3 NSSA external type 1,
       O3 N2 - OSPFv3 NSSA external type2, B - Other BGP Routes,
       B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
       I L2 - IS-IS level 2, A B - BGP Aggregate,
       A O - OSPF Summary, NG - Nexthop Group Static Route,
       V - VXLAN Control Service, M - Martian,
       DH - DHCP client installed default route,
       DP - Dynamic Policy Route, L - VRF Leaked,
       G  - gRIBI, RC - Route Cache Route,
       CL - CBF Leaked Route

Gateway of last resort is not set

 B I      10.0.20.8/32 [200/0]
           via VTEP 10.0.4.1 VNI 10001 router-mac ce:ec:f4:6c:d0:d1
 B I      10.0.20.42/32 [200/0]
           via VTEP 10.0.4.1 VNI 10001 router-mac ce:ec:f4:6c:d0:d1
 C        10.0.20.0/24
           directly connected, Vlan1001


VRF: 10002
Source Codes:
       C - connected, S - static, K - kernel,
       O - OSPF, O IA - OSPF inter area, O E1 - OSPF external type 1,
       O E2 - OSPF external type 2, O N1 - OSPF NSSA external type 1,
       O N2 - OSPF NSSA external type2, O3 - OSPFv3,
       O3 IA - OSPFv3 inter area, O3 E1 - OSPFv3 external type 1,
       O3 E2 - OSPFv3 external type 2,
       O3 N1 - OSPFv3 NSSA external type 1,
       O3 N2 - OSPFv3 NSSA external type2, B - Other BGP Routes,
       B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
       I L2 - IS-IS level 2, A B - BGP Aggregate,
       A O - OSPF Summary, NG - Nexthop Group Static Route,
       V - VXLAN Control Service, M - Martian,
       DH - DHCP client installed default route,
       DP - Dynamic Policy Route, L - VRF Leaked,
       G  - gRIBI, RC - Route Cache Route,
       CL - CBF Leaked Route

Gateway of last resort is not set



VRF: 10007
Source Codes:
       C - connected, S - static, K - kernel,
       O - OSPF, O IA - OSPF inter area, O E1 - OSPF external type 1,
       O E2 - OSPF external type 2, O N1 - OSPF NSSA external type 1,
       O N2 - OSPF NSSA external type2, O3 - OSPFv3,
       O3 IA - OSPFv3 inter area, O3 E1 - OSPFv3 external type 1,
       O3 E2 - OSPFv3 external type 2,
       O3 N1 - OSPFv3 NSSA external type 1,
       O3 N2 - OSPFv3 NSSA external type2, B - Other BGP Routes,
       B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
       I L2 - IS-IS level 2, A B - BGP Aggregate,
       A O - OSPF Summary, NG - Nexthop Group Static Route,
       V - VXLAN Control Service, M - Martian,
       DH - DHCP client installed default route,
       DP - Dynamic Policy Route, L - VRF Leaked,
       G  - gRIBI, RC - Route Cache Route,
       CL - CBF Leaked Route

Gateway of last resort is not set

 C        10.0.22.0/24
           directly connected, Vlan1007

Arista: resolved MAC addresses
Code:
arista7060cx#show ip arp vrf all

VRF: 10001
Address         Age (sec)  Hardware Addr   Interface
10.0.20.8               -  0aa9.2c3c.b7a0  Vlan1001, Vxlan1
10.0.20.42              -  bc24.1126.9cbb  Vlan1001, Vxlan1

VRF: 10002
Address         Age (sec)  Hardware Addr   Interface

VRF: 10007
Address         Age (sec)  Hardware Addr   Interface


VRF: default
Address         Age (sec)  Hardware Addr   Interface
5.133.66.1        0:00:00  0000.5e00.0142  Ethernet34
5.133.66.29       3:30:06  bc24.113e.fc83  Ethernet34
5.133.66.253      3:44:31  2c6b.f5a5.27c6  Ethernet34
5.133.66.254      1:30:04  28c0.da07.b7cc  Ethernet34
10.0.4.1          0:00:01  ec0d.9a43.fb34  Vlan3, Ethernet1/1

Arista: ping to VM fail
Code:
arista7060cx#ping vrf 10001 10.0.20.8
PING 10.0.20.8 (10.0.20.8) 72(100) bytes of data.

--- 10.0.20.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Proxmox: node (10.0.4.1) does receive ICMP:
Code:
14:04:03.274570 vrfvx_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 1, length 80
14:04:03.274570 vrfbr_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 1, length 80
14:04:03.284679 vrfvx_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 2, length 80
14:04:03.284679 vrfbr_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 2, length 80
14:04:03.294767 vrfvx_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 3, length 80
14:04:03.294767 vrfbr_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 3, length 80
14:04:03.304845 vrfvx_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 4, length 80
14:04:03.304845 vrfbr_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 4, length 80
14:04:03.314925 vrfvx_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 5, length 80
14:04:03.314925 vrfbr_z10001 In  IP 10.0.20.1 > 10.0.20.8: ICMP echo request, id 43833, seq 5, length 80

Proxmox: ping from node (10.0.4.1) to VM:
Code:
root@proxmox1-1:/etc/pve/sdn# ping 10.0.20.8
PING 10.0.20.8 (10.0.20.8) 56(84) bytes of data.
64 bytes from 10.0.20.8: icmp_seq=1 ttl=64 time=0.332 ms
64 bytes from 10.0.20.8: icmp_seq=2 ttl=64 time=0.255 ms
64 bytes from 10.0.20.8: icmp_seq=3 ttl=64 time=0.316 ms
 
at minimum, if you use your arista as exit-node, (advertising the default route from arista)

you should remove
"exitnodes proxmox1-4,proxmox1-3,proxmox1-1"


and peer with your arista router

"peers proxmox1ip, proxmox2ip, proxmox3ip, aristaip"


snat should be done on your arista too
 
Have you checked the VM firewall? Does Arista have the type 2 and type 3 routes?

Yes, firewall is disabled on proxmox node and VM. Arista does receive Type-2, Type-3 and Type-5 routes from proxmox node (10.0.4.1). Everything looks correct to me.

imet routes:
Code:
arista7060cx(config-router-bgp-vrf-10007)#show bgp neighbors 10.0.4.1 evpn received-routes route-type imet

          Network                Next Hop              Metric  LocPref Weight  Path
 * >      RD: 10.0.4.1:8 imet 10.0.4.1
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:10 imet 10.0.4.1
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:12 imet 10.0.4.1
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:14 imet 10.0.4.1
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:16 imet 10.0.4.1
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:18 imet 10.0.4.1
                                 10.0.4.1              -       100     0       i

mac-ip routes:
Code:
arista7060cx(config-router-bgp-vrf-10007)#show bgp neighbors 10.0.4.1 evpn received-routes route-type mac-ip

          Network                Next Hop              Metric  LocPref Weight  Path
 * >      RD: 10.0.4.1:8 mac-ip 0aa9.2c3c.b7a0
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip 0aa9.2c3c.b7a0 10.0.20.8
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip 0aa9.2c3c.b7a0 fe80::8a9:2cff:fe3c:b7a0
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip 1afb.635c.774f
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip 1afb.635c.774f fe80::18fb:63ff:fe5c:774f
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip 4a70.56de.7973
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip 4a70.56de.7973 fe80::4870:56ff:fede:7973
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip 5205.b645.ad4d
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip bc24.1126.9cbb
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip bc24.1126.9cbb 10.0.20.42
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip bc24.1128.99d8
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip bc24.1128.99d8 2001:1ab9:f002:2::6e
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip bc24.1128.99d8 fe80::be24:11ff:fe28:99d8
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip e6cb.1735.c2d2
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip e6cb.1735.c2d2 2001:1ab9:f002:2::73
                                 10.0.4.1              -       100     0       i
 * >      RD: 10.0.4.1:8 mac-ip e6cb.1735.c2d2 fe80::e4cb:17ff:fe35:c2d2
                                 10.0.4.1              -       100     0       i
 
at minimum, if you use your arista as exit-node, (advertising the default route from arista)

you should remove
"exitnodes proxmox1-4,proxmox1-3,proxmox1-1"


and peer with your arista router

"peers proxmox1ip, proxmox2ip, proxmox3ip, aristaip"


snat should be done on your arista too
At the moment, I do not have the Arista as an exit node; I haven’t figured out how to advertise the default route from the Arista yet.
Should it be Type-5 route for each VRF?



I already have aristaip (192.168.10.1) in vpn controller peer, along with static route to reach it.

Code:
evpn: evpn1
        asn 65000
        peers 10.0.4.1,10.0.4.3,10.0.4.4,192.168.10.1
 
Last edited:
At the moment, I do not have the Arista as an exit node; I haven’t figured out how to advertise the default route from the Arista yet.
Should it be Type-5 route for each VRF?
yes, you need to announce a type-5 0.0.0.0 for each vrf.
you need to setup a symetric evpn config on your arista
https://www.youtube.com/watch?v=z26zM-GF4WM

It's really the way to do it (I'm running arista in production), as you have the chance to have a switch with a correct evpn implementation.
 
yes, you need to announce a type-5 0.0.0.0 for each vrf.
you need to setup a symetric evpn config on your arista
https://www.youtube.com/watch?v=z26zM-GF4WM

It's really the way to do it (I'm running arista in production), as you have the chance to have a switch with a correct evpn implementation.
Thank you @aderumier, that was very helpful. My setup is Symmetric IRB without MLAG. After some digging I found that Arista wasn’t advertising Type-5 default routes and Type-2 routes for the SVIs. Now it does, but I still can’t ping the VMs. Also, Arista is not listed as an “Exit node” for EVPN zones in the Proxmox GUI.

The only way I managed to advertise default routes was with the following configuration — maybe there is a dedicated configuration directive for this?

Code:
ip route vrf 10001 0.0.0.0/0 Null0
!
router bgp 65000
    vrf 10001
    redistribute static

Here are the routes that Arista (192.168.10.1) is currently advertising:
Code:
arista7060cx(config)#show bgp neighbors 10.0.4.1 evpn advertised-routes
 * >      RD: 192.168.10.1:1001 mac-ip 001c.7300.0001 10.0.20.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1002 mac-ip 001c.7300.0001 10.0.21.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1007 mac-ip 001c.7300.0001 10.0.22.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1009 mac-ip 001c.7300.0001 10.0.23.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1013 mac-ip 001c.7300.0001 10.0.24.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1001 imet 192.168.10.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1002 imet 192.168.10.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1007 imet 192.168.10.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1009 imet 192.168.10.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 192.168.10.1:1013 imet 192.168.10.1
                                 192.168.10.1          -       100     0       i
 * >      RD: 65000:10001 ip-prefix 0.0.0.0/0
                                 192.168.10.1          -       100     0       ?
 * >      RD: 65000:10002 ip-prefix 0.0.0.0/0
                                 192.168.10.1          -       100     0       ?
 * >      RD: 65000:10007 ip-prefix 0.0.0.0/0
                                 192.168.10.1          -       100     0       ?
 * >      RD: 65000:10009 ip-prefix 0.0.0.0/0
                                 192.168.10.1          -       100     0       ?
 * >      RD: 65000:10013 ip-prefix 0.0.0.0/0
                                 192.168.10.1          -       100     0       ?
 * >      RD: 65000:10001 ip-prefix 10.0.20.0/24
                                 192.168.10.1          -       100     0       i
 * >      RD: 65000:10002 ip-prefix 10.0.21.0/24
                                 192.168.10.1          -       100     0       i
 * >      RD: 65000:10007 ip-prefix 10.0.22.0/24
                                 192.168.10.1          -       100     0       i
 * >      RD: 65000:10009 ip-prefix 10.0.23.0/24
                                 192.168.10.1          -       100     0       i
 * >      RD: 65000:10013 ip-prefix 10.0.24.0/24
                                 192.168.10.1          -       100     0       i

Proxmox does see Arista as VTEP (192.168.10.1):
Code:
root@proxmox1-1:~# vtysh -c "show evpn next-hops vni all"

VNI 10001 #Next-Hops 4

IP              RMAC
192.168.10.1    00:1c:73:00:00:01
::ffff:a00:403  ee:ea:d3:0e:9c:2f
::ffff:a00:404  12:6e:ec:45:14:5d
10.0.4.4        12:6e:ec:45:14:5d

VNI 10002 #Next-Hops 1

IP              RMAC
192.168.10.1    00:1c:73:00:00:01

VNI 10007 #Next-Hops 1

IP              RMAC
192.168.10.1    00:1c:73:00:00:01

VNI 10009 #Next-Hops 1

IP              RMAC
192.168.10.1    00:1c:73:00:00:01


Are you willing to share your configuration? I could also share mine — maybe you will notice something I’m missing.
 
Last edited:
Are you advertising bgp routes to l2vpn? Without that, no type 5 routes are generated.

evpn
!
vrf DMZ
vni 14000
rd 10.127.1.1:14000
route-target 65100:14000 both
advertise ipv4 bgp
advertise ipv6 bgp
exit
!
vrf ACCESS
vni 15000
rd 10.127.1.1:15000
route-target 65100:15000 both
advertise ipv4 bgp
advertise ipv6 bgp
exit
exit
!
 
Ah, I completely misunderstood the purpose of "exit nodes," but now it's clear. However, I still can't reach the VMs from the Arista switch. I think my problem is related to the MTU. It looks like the Arista is forcing an MTU of 1330 somewhere, even though the underlay interfaces have their MTU set to 1550.

Code:
arista7060cx#ping vrf 10001 10.0.20.8 size 1400 df-bit
PING 10.0.20.8 (10.0.20.8) 1372(1400) bytes of data.
ping: local error: message too long, mtu=1300
ping: local error: message too long, mtu=1300
ping: local error: message too long, mtu=1300
ping: local error: message too long, mtu=1300
ping: local error: message too long, mtu=1300

Anyway, you really helped me to understand evpn better @aderumier, thank you.
 
Last edited: