SDN with EVPN Controller

K

kemeris

Member
Nov 23, 2021
27
1
8
46
Hello,

I have been exploring SDN. Right now, I have an cluster with three nodes, VXLAN is working fine between nodes.

My problem is routing from router (static route, router does not support SDN) to PVE node. I can ping PVE node from router but can't ping VM or VNet's subnet gateway. Firewall is disabled. I would appreciate any help.

controllers.cfg
Code: 
evpn: evpn1
        asn 65000
        peers 10.0.4.1,10.0.4.3,10.0.4.4

zones.cfg
Code: 
evpn: z100001
        controller evpn1
        vrf-vxlan 100001
        advertise-subnets 1
        exitnodes proxmox1-4,proxmox1-1,proxmox1-3
        exitnodes-primary proxmox1-4
        ipam pve
        mac 7E:11:50:67:46:63
        mtu 1500

vnets.cfg
Code: 
vnet: v500001
        zone z100001
        tag 500001

subnets.cfg
Code: 
subnet: z100001-10.0.20.0-24
        vnet v500001
        gateway 10.0.20.1


Code: 
external router----->route add 10.0.20.0/24 gw 10.0.4.4--->exitnodes(proxmox1-4)--->vnet v500001(10.0.20.1)---->vm(10.0.20.10)

1704468677462.png





Thank you for reading.
 
Last edited:
Thanks spirit. Changed to 1 exit-node, updated PVE 8.0.4 to 8.1.3 (frr version is 8.5.1) but nothing changed. Node 10.0.4.4 can ping and SSH into VM (10.0.20.10), sysctl net.ipv4.tcp_l3mdev_accept=0. Link between router and nodes are in vlan 3 with MTU 1550, could it be a problem?

zones.cfg:
Code: 
evpn: z100001
        controller evpn1
        vrf-vxlan 100001
        advertise-subnets 1
        disable-arp-nd-suppression 1
        exitnodes proxmox1-4
        exitnodes-primary proxmox1-4
        ipam pve
        mac 7E:11:50:67:46:63
        mtu 1500

This is routing table of 10.0.4.4 node:
Code: 
root@proxmox1-4:/etc/pve/sdn# ip r s
default via 5.133.66.1 dev vmbr1 proto kernel onlink
default nhid 56 proto bgp metric 20
        nexthop via 10.0.4.1 dev vrfbr_z100002 weight 1 onlink
        nexthop via 10.0.4.3 dev vrfbr_z100002 weight 1 onlink
5.133.66.0/24 dev vmbr1 proto kernel scope link src 5.133.66.25
10.0.4.0/22 dev bond0_25G.3 proto kernel scope link src 10.0.4.4
10.0.8.0/22 dev bond0_25G.4 proto kernel scope link src 10.0.8.4
10.0.12.0/22 dev eno7_50G.5 proto kernel scope link src 10.0.12.4
10.0.20.0/24 nhid 197 dev v500001 proto bgp metric 20
10.0.20.8 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.9 nhid 202 via 10.0.4.3 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.15 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.16 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.17 nhid 202 via 10.0.4.3 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.19 nhid 202 via 10.0.4.3 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.20 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.31 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.33 nhid 202 via 10.0.4.3 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.21.0/24 nhid 27 dev v500002 proto bgp metric 20
10.0.24.0/24 nhid 101 dev v500003 proto bgp metric 20
10.0.100.0/23 nhid 75 dev v500008 proto bgp metric 20
 
can you send result of

vtysh -c "sh bgp l2evpn evpn"

?

and content of /etc/frr/frr.conf of each nodes ?


also, try to remove "exitnodes-primary" for the test, it's only needed with more than 1 exit-node.


This is strange that you still have

"
default nhid 56 proto bgp metric 20
nexthop via 10.0.4.1 dev vrfbr_z100002 weight 1 onlink
nexthop via 10.0.4.3 dev vrfbr_z100002 weight 1 onlink
"

This is the bug with multiple exit-nodes currently, where a filtering of default route is broken.
(I have sent fix last month, but it's not yet available in repos)
 
sysctl net.ipv4.tcp_l3mdev_accept=0
Click to expand...
Sorry for my mistype. tcp_l3mdev_accept is actually enabled (exit-nodes-local-routing disabled). Without it node (10.0.4.4) can't access local VM1 (10.0.20.10).


also, try to remove "exitnodes-primary" for the test, it's only needed with more than 1 exit-node.
Click to expand...
It's not possible to remove it anymore via GUI after update to PVE v8.1.3. I have removed it from zones.cfg and applied changes, but nothing changed.

1704622655474.png

I have simplified my network setup a little, removed one evpn zone, restarted networking. Now I can access ONLY this one VM (10.0.20.10) from router. Other VM's on same node and zone is not reachable.

vtysh -c "sh bgp l2vpn evpn" on node 10.0.4.4:
Code: 
BGP table version is 6, local router ID is 10.0.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]


   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.4.1:2
 *>i[5]:[0]:[0]:[0.0.0.0]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100002 ET:8 Rmac:56:f5:fd:84:57:d9
 *>i[5]:[0]:[0]:[::] 10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100002 ET:8 Rmac:56:f5:fd:84:57:d9
Route Distinguisher: 10.0.4.1:3
 *>i[5]:[0]:[0]:[0.0.0.0]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100003 ET:8 Rmac:9a:fa:e3:32:f5:1e
 *>i[5]:[0]:[0]:[::] 10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100003 ET:8 Rmac:9a:fa:e3:32:f5:1e
Route Distinguisher: 10.0.4.1:4
 *>i[3]:[0]:[32]:[10.0.4.1]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500003 ET:8
Route Distinguisher: 10.0.4.1:5
 *>i[2]:[0]:[48]:[0a:a9:2c:3c:b7:a0]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[1a:fb:63:5c:77:4f]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[4a:70:56:de:79:73]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[4a:70:56:de:79:73]:[32]:[10.0.20.31]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100001 RT:65000:500001 ET:8 Rmac:fa:59:d4:2a:a8:c5
 *>i[2]:[0]:[48]:[76:c4:c1:91:aa:c3]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[76:c4:c1:91:aa:c3]:[32]:[10.0.20.20]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100001 RT:65000:500001 ET:8 Rmac:fa:59:d4:2a:a8:c5
 *>i[2]:[0]:[48]:[e6:cb:17:35:c2:d2]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[e6:cb:17:35:c2:d2]:[32]:[10.0.20.16]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100001 RT:65000:500001 ET:8 Rmac:fa:59:d4:2a:a8:c5
 *>i[3]:[0]:[32]:[10.0.4.1]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
Route Distinguisher: 10.0.4.1:6
 *>i[5]:[0]:[24]:[10.0.20.0]
                    10.0.4.1(proxmox1-1)
                                             0    100      0 ?
                    RT:65000:100001 ET:8 Rmac:fa:59:d4:2a:a8:c5
Route Distinguisher: 10.0.4.1:7
 *>i[3]:[0]:[32]:[10.0.4.1]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500002 ET:8
Route Distinguisher: 10.0.4.3:2
 *>i[5]:[0]:[0]:[0.0.0.0]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100002 ET:8 Rmac:f6:36:76:14:44:a1
 *>i[5]:[0]:[0]:[::] 10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100002 ET:8 Rmac:f6:36:76:14:44:a1
Route Distinguisher: 10.0.4.3:3
 *>i[5]:[0]:[0]:[0.0.0.0]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100003 ET:8 Rmac:a2:dc:16:c1:6f:cd
 *>i[5]:[0]:[0]:[::] 10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100003 ET:8 Rmac:a2:dc:16:c1:6f:cd
Route Distinguisher: 10.0.4.3:4
 *>i[3]:[0]:[32]:[10.0.4.3]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500003 ET:8
Route Distinguisher: 10.0.4.3:5
 *>i[2]:[0]:[48]:[32:62:d4:b9:d6:e3]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[52:05:b6:45:ad:4d]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[52:05:b6:45:ad:4d]:[32]:[10.0.20.19]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100001 RT:65000:500001 ET:8 Rmac:96:d6:ef:8f:d0:a4
 *>i[2]:[0]:[48]:[ca:d2:d3:a0:b3:ce]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[d6:9f:26:a2:04:40]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[3]:[0]:[32]:[10.0.4.3]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
Route Distinguisher: 10.0.4.3:6
 *>i[5]:[0]:[24]:[10.0.20.0]
                    10.0.4.3(proxmox1-3)
                                             0    100      0 ?
                    RT:65000:100001 ET:8 Rmac:96:d6:ef:8f:d0:a4
Route Distinguisher: 10.0.4.3:7
 *>i[2]:[0]:[48]:[bc:24:11:f5:32:01]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500002 ET:8 MM:1
 *>i[2]:[0]:[48]:[bc:24:11:f5:32:01]:[32]:[10.0.21.2]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100002 RT:65000:500002 ET:8 MM:1 Rmac:f6:36:76:14:44:a1
 *>i[3]:[0]:[32]:[10.0.4.3]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500002 ET:8
Route Distinguisher: 10.0.4.4:2
 *> [5]:[0]:[0]:[0.0.0.0]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [5]:[0]:[0]:[::] 10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100001 Rmac:52:51:41:96:b3:70
Route Distinguisher: 10.0.4.4:3
 *> [5]:[0]:[0]:[0.0.0.0]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100002 Rmac:f6:df:8f:9c:a0:16
 *> [5]:[0]:[0]:[::] 10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100002 Rmac:f6:df:8f:9c:a0:16
Route Distinguisher: 10.0.4.4:4
 *> [5]:[0]:[0]:[0.0.0.0]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100003 Rmac:9a:a6:c1:68:2b:43
 *> [5]:[0]:[0]:[::] 10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100003 Rmac:9a:a6:c1:68:2b:43
Route Distinguisher: 10.0.4.4:5
 *> [3]:[0]:[32]:[10.0.4.4]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500003
Route Distinguisher: 10.0.4.4:6
 *> [2]:[0]:[48]:[12:58:ed:f9:9f:c0]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
 *> [2]:[0]:[48]:[12:58:ed:f9:9f:c0]:[32]:[10.0.20.23]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [2]:[0]:[48]:[52:5a:13:a8:73:81]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
 *> [2]:[0]:[48]:[52:5a:13:a8:73:81]:[32]:[10.0.20.32]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [2]:[0]:[48]:[9e:c1:fa:5a:f3:07]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
 *> [2]:[0]:[48]:[9e:c1:fa:5a:f3:07]:[32]:[10.0.20.10]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [2]:[0]:[48]:[de:d6:ab:7b:95:f1]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
 *> [2]:[0]:[48]:[de:d6:ab:7b:95:f1]:[32]:[10.0.20.13]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [3]:[0]:[32]:[10.0.4.4]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
Route Distinguisher: 10.0.4.4:7
 *> [3]:[0]:[32]:[10.0.4.4]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500002


Displayed 48 out of 48 total prefixes


/etc/frr/frr.conf of node 10.0.4.4:
Code: 
frr version 8.5.1
frr defaults datacenter
hostname proxmox1-4
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_z100001
 vni 100001
 ip route 10.0.21.0/24 null0
 ip route 10.0.24.0/24 null0
exit-vrf
!
vrf vrf_z100002
 vni 100002
 ip route 10.0.20.0/24 null0
 ip route 10.0.24.0/24 null0
exit-vrf
!
vrf vrf_z100003
 vni 100003
 ip route 10.0.20.0/24 null0
 ip route 10.0.21.0/24 null0
exit-vrf
!
router bgp 65000
 bgp router-id 10.0.4.4
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 10.0.4.1 peer-group VTEP
 neighbor 10.0.4.3 peer-group VTEP
 !
 address-family ipv4 unicast
  import vrf vrf_z100001
  import vrf vrf_z100002
  import vrf vrf_z100003
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_z100001
  import vrf vrf_z100002
  import vrf vrf_z100003
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_z100001
 bgp router-id 10.0.4.4
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
router bgp 65000 vrf vrf_z100002
 bgp router-id 10.0.4.4
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
router bgp 65000 vrf vrf_z100003
 bgp router-id 10.0.4.4
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match evpn vni 100002
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN deny 2
 match evpn vni 100001
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 3
exit
!
route-map MAP_VTEP_OUT permit 1
 match evpn vni 100003
 match evpn route-type prefix
 set metric 200
exit
!
route-map MAP_VTEP_OUT permit 2
exit
!
line vty
 
Last edited:
/etc/frr/frr.conf of node 10.0.4.3:
Code: 
frr version 8.5.1

frr defaults datacenter

hostname proxmox1-3

log syslog informational

service integrated-vtysh-config

!

!

vrf vrf_z100001

 vni 100001

exit-vrf

!

vrf vrf_z100002

 vni 100002

 ip route 10.0.20.0/24 null0

 ip route 10.0.24.0/24 null0

exit-vrf

!

vrf vrf_z100003

 vni 100003

 ip route 10.0.20.0/24 null0

 ip route 10.0.21.0/24 null0

exit-vrf

!

router bgp 65000

 bgp router-id 10.0.4.3

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 no bgp default ipv4-unicast

 coalesce-time 1000

 neighbor VTEP peer-group

 neighbor VTEP remote-as 65000

 neighbor VTEP bfd

 neighbor 10.0.4.1 peer-group VTEP

 neighbor 10.0.4.4 peer-group VTEP

 !

 address-family ipv4 unicast

  import vrf vrf_z100002

  import vrf vrf_z100003

 exit-address-family

 !

 address-family ipv6 unicast

  import vrf vrf_z100002

  import vrf vrf_z100003

 exit-address-family

 !

 address-family l2vpn evpn

  neighbor VTEP route-map MAP_VTEP_IN in

  neighbor VTEP route-map MAP_VTEP_OUT out

  neighbor VTEP activate

  advertise-all-vni

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100001

 bgp router-id 10.0.4.3

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  advertise ipv4 unicast

  advertise ipv6 unicast

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100002

 bgp router-id 10.0.4.3

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  default-originate ipv4

  default-originate ipv6

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100003

 bgp router-id 10.0.4.3

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  default-originate ipv4

  default-originate ipv6

 exit-address-family

exit

!

route-map MAP_VTEP_IN permit 1

exit

!

route-map MAP_VTEP_OUT permit 1

 match evpn vni 100003

 match evpn route-type prefix

 set metric 200

exit

!

route-map MAP_VTEP_OUT permit 2

 match evpn vni 100002

 match evpn route-type prefix

 set metric 200

exit

!

route-map MAP_VTEP_OUT permit 3

exit

!

line vty



/etc/frr/frr.conf of node 10.0.4.1:
Code: 
frr version 8.5.1

frr defaults datacenter

hostname proxmox1-1

log syslog informational

service integrated-vtysh-config

!

!

vrf vrf_z100001

 vni 100001

exit-vrf

!

vrf vrf_z100002

 vni 100002

 ip route 10.0.20.0/24 null0

 ip route 10.0.24.0/24 null0

exit-vrf

!

vrf vrf_z100003

 vni 100003

 ip route 10.0.20.0/24 null0

 ip route 10.0.21.0/24 null0

exit-vrf

!

router bgp 65000

 bgp router-id 10.0.4.1

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 no bgp default ipv4-unicast

 coalesce-time 1000

 neighbor VTEP peer-group

 neighbor VTEP remote-as 65000

 neighbor VTEP bfd

 neighbor 10.0.4.3 peer-group VTEP

 neighbor 10.0.4.4 peer-group VTEP

 !

 address-family ipv4 unicast

  import vrf vrf_z100002

  import vrf vrf_z100003

 exit-address-family

 !

 address-family ipv6 unicast

  import vrf vrf_z100002

  import vrf vrf_z100003

 exit-address-family

 !

 address-family l2vpn evpn

  neighbor VTEP route-map MAP_VTEP_IN in

  neighbor VTEP route-map MAP_VTEP_OUT out

  neighbor VTEP activate

  advertise-all-vni

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100001

 bgp router-id 10.0.4.1

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  advertise ipv4 unicast

  advertise ipv6 unicast

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100002

 bgp router-id 10.0.4.1

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  default-originate ipv4

  default-originate ipv6

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100003

 bgp router-id 10.0.4.1

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  default-originate ipv4

  default-originate ipv6

 exit-address-family

exit

!

route-map MAP_VTEP_IN deny 1

 match evpn vni 100003

 match evpn route-type prefix

exit

!

route-map MAP_VTEP_IN permit 2

exit

!

route-map MAP_VTEP_OUT permit 1

 match evpn vni 100002

 match evpn route-type prefix

 set metric 200

exit

!

route-map MAP_VTEP_OUT permit 2

exit

!

line vty


spirit said:
This is the bug with multiple exit-nodes currently, where a filtering of default route is broken.
(I have sent fix last month, but it's not yet available in repos)
Click to expand...
If there is nothing I can do, I will wait for bugfix.
 
Last edited:
mmm, this is strange, I still see others others as exit-nodes in their frr.conf

" default-originate ipv4"

(so you are still having the bug with multiple exit-nodes)

Are you sure to have correctly apply/reload the sdn configuration ?



I'll look at the gui for exit-node primary, it shouldn't be mandatory. (but I don't think it's a problem for your bug here)


About "exit-nodes-local-routing", you can remove it, it's only needed if you want to reach the vm ip from the hypervisor ip itself.
 
I am sorry for my late response.

spirit said:
mmm, this is strange, I still see others others as exit-nodes in their frr.conf

" default-originate ipv4"

(so you are still having the bug with multiple exit-nodes)

Are you sure to have correctly apply/reload the sdn configuration ?



I'll look at the gui for exit-node primary, it shouldn't be mandatory. (but I don't think it's a problem for your bug here)


About "exit-nodes-local-routing", you can remove it, it's only needed if you want to reach the vm ip from the hypervisor ip itself.
Click to expand...
Yes, I removed it from one zone only. Now I removed it from all zones and there is no more "default-originate ipv4" in frr.conf and ECMP routes in routing table.

spirit said:
About "exit-nodes-local-routing", you can remove it, it's only needed if you want to reach the vm ip from the hypervisor ip itself.
Click to expand...

Yes, I know that from your post in other threads.

I think I have found why I can't access (from router) some VM IP's while can access others within same Zone on same node.

All VM's I can't access had two interfaces with one public IP/gateway attached to node's bridge.
IP=10.0.20.32/24 gateway=none -> EVPN
IP=5.133.66.48/24 gateway=5.133.66.1 -> Node bridge

Once I switched gateway (10.0.20.1) to evpn interface, I can access VM from router. Will I be able to use public IP as VM's default gateway if I will move public IP into separate Vnet subnet?
 
kemeris said:
Once I switched gateway (10.0.20.1) to evpn interface, I can access VM from router. Will I be able to use public IP as VM's default gateway if I will move public IP into separate Vnet subnet?
Click to expand...
you can only have 1 default gw in your vm, so it must be on public ip.

if you need to to reach private ip, from another private network, you need to a static route in your vm like:

"ip route add 10.0.0.0/8 gw 10.0.20.1"
 
spirit said:
you can only have 1 default gw in your vm, so it must be on public ip.

if you need to to reach private ip, from another private network, you need to a static route in your vm like:

"ip route add 10.0.0.0/8 gw 10.0.20.1"
Click to expand...

Thank you spirit for your time, I really appreciate that.

Can confirm "ip route add 10.0.0.0/8 via 10.0.20.1" works, I am able to reach private/public IPs within VNet from router.
Had to set
Code: 
net.ipv4.conf.default.rp_filter = 0
and
Code: 
net.ipv4.conf.all.rp_filter = 0
as your mentioned in this thread.

As I understand it's not possible to route traffic from certain VM's in VNet to PVE node (via private IP)? At the moment I simply attach secondary VM interface to bridge on PVE node to run automation tasks to PVE API.
 
You must log in or register to reply here.
Top Bottom