SDN with EVPN Controller

kemeris · Jan 5, 2024

Hello,

I have been exploring SDN. Right now, I have an cluster with three nodes, VXLAN is working fine between nodes.

My problem is routing from router (static route, router does not support SDN) to PVE node. I can ping PVE node from router but can't ping VM or VNet's subnet gateway. Firewall is disabled. I would appreciate any help.

controllers.cfg

Code:

evpn: evpn1
        asn 65000
        peers 10.0.4.1,10.0.4.3,10.0.4.4

zones.cfg

Code:

evpn: z100001
        controller evpn1
        vrf-vxlan 100001
        advertise-subnets 1
        exitnodes proxmox1-4,proxmox1-1,proxmox1-3
        exitnodes-primary proxmox1-4
        ipam pve
        mac 7E:11:50:67:46:63
        mtu 1500

vnets.cfg

Code:

vnet: v500001
        zone z100001
        tag 500001

subnets.cfg

Code:

subnet: z100001-10.0.20.0-24
        vnet v500001
        gateway 10.0.20.1

Code:

external router----->route add 10.0.20.0/24 gw 10.0.4.4--->exitnodes(proxmox1-4)--->vnet v500001(10.0.20.1)---->vm(10.0.20.10)

Thank you for reading.

spirit · Jan 6, 2024

Hi, can you try with only 1 exit-node ? (your setup seem correct)

They are currently a bug with multiple exit-nodes with pve8 && frr version, I have sent patches last month, but they are not yet applied.

kemeris · Jan 7, 2024

Thanks spirit. Changed to 1 exit-node, updated PVE 8.0.4 to 8.1.3 (frr version is 8.5.1) but nothing changed. Node 10.0.4.4 can ping and SSH into VM (10.0.20.10), sysctl net.ipv4.tcp_l3mdev_accept=0. Link between router and nodes are in vlan 3 with MTU 1550, could it be a problem?

zones.cfg:

Code:

evpn: z100001
        controller evpn1
        vrf-vxlan 100001
        advertise-subnets 1
        disable-arp-nd-suppression 1
        exitnodes proxmox1-4
        exitnodes-primary proxmox1-4
        ipam pve
        mac 7E:11:50:67:46:63
        mtu 1500

This is routing table of 10.0.4.4 node:

Code:

root@proxmox1-4:/etc/pve/sdn# ip r s
default via 5.133.66.1 dev vmbr1 proto kernel onlink
default nhid 56 proto bgp metric 20
        nexthop via 10.0.4.1 dev vrfbr_z100002 weight 1 onlink
        nexthop via 10.0.4.3 dev vrfbr_z100002 weight 1 onlink
5.133.66.0/24 dev vmbr1 proto kernel scope link src 5.133.66.25
10.0.4.0/22 dev bond0_25G.3 proto kernel scope link src 10.0.4.4
10.0.8.0/22 dev bond0_25G.4 proto kernel scope link src 10.0.8.4
10.0.12.0/22 dev eno7_50G.5 proto kernel scope link src 10.0.12.4
10.0.20.0/24 nhid 197 dev v500001 proto bgp metric 20
10.0.20.8 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.9 nhid 202 via 10.0.4.3 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.15 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.16 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.17 nhid 202 via 10.0.4.3 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.19 nhid 202 via 10.0.4.3 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.20 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.31 nhid 201 via 10.0.4.1 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.20.33 nhid 202 via 10.0.4.3 dev vrfbr_z100001 proto bgp metric 20 onlink
10.0.21.0/24 nhid 27 dev v500002 proto bgp metric 20
10.0.24.0/24 nhid 101 dev v500003 proto bgp metric 20
10.0.100.0/23 nhid 75 dev v500008 proto bgp metric 20

spirit · Jan 7, 2024

can you send result of

vtysh -c "sh bgp l2evpn evpn"

?

and content of /etc/frr/frr.conf of each nodes ?

also, try to remove "exitnodes-primary" for the test, it's only needed with more than 1 exit-node.

This is strange that you still have

"
default nhid 56 proto bgp metric 20
nexthop via 10.0.4.1 dev vrfbr_z100002 weight 1 onlink
nexthop via 10.0.4.3 dev vrfbr_z100002 weight 1 onlink
"

This is the bug with multiple exit-nodes currently, where a filtering of default route is broken.
(I have sent fix last month, but it's not yet available in repos)

kemeris · Jan 7, 2024

sysctl net.ipv4.tcp_l3mdev_accept=0

Sorry for my mistype. tcp_l3mdev_accept is actually enabled (exit-nodes-local-routing disabled). Without it node (10.0.4.4) can't access local VM1 (10.0.20.10).

also, try to remove "exitnodes-primary" for the test, it's only needed with more than 1 exit-node.

It's not possible to remove it anymore via GUI after update to PVE v8.1.3. I have removed it from zones.cfg and applied changes, but nothing changed.

I have simplified my network setup a little, removed one evpn zone, restarted networking. Now I can access ONLY this one VM (10.0.20.10) from router. Other VM's on same node and zone is not reachable.

vtysh -c "sh bgp l2vpn evpn" on node 10.0.4.4:

Code:

BGP table version is 6, local router ID is 10.0.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]


   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.4.1:2
 *>i[5]:[0]:[0]:[0.0.0.0]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100002 ET:8 Rmac:56:f5:fd:84:57:d9
 *>i[5]:[0]:[0]:[::] 10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100002 ET:8 Rmac:56:f5:fd:84:57:d9
Route Distinguisher: 10.0.4.1:3
 *>i[5]:[0]:[0]:[0.0.0.0]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100003 ET:8 Rmac:9a:fa:e3:32:f5:1e
 *>i[5]:[0]:[0]:[::] 10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100003 ET:8 Rmac:9a:fa:e3:32:f5:1e
Route Distinguisher: 10.0.4.1:4
 *>i[3]:[0]:[32]:[10.0.4.1]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500003 ET:8
Route Distinguisher: 10.0.4.1:5
 *>i[2]:[0]:[48]:[0a:a9:2c:3c:b7:a0]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[1a:fb:63:5c:77:4f]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[4a:70:56:de:79:73]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[4a:70:56:de:79:73]:[32]:[10.0.20.31]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100001 RT:65000:500001 ET:8 Rmac:fa:59:d4:2a:a8:c5
 *>i[2]:[0]:[48]:[76:c4:c1:91:aa:c3]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[76:c4:c1:91:aa:c3]:[32]:[10.0.20.20]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100001 RT:65000:500001 ET:8 Rmac:fa:59:d4:2a:a8:c5
 *>i[2]:[0]:[48]:[e6:cb:17:35:c2:d2]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[e6:cb:17:35:c2:d2]:[32]:[10.0.20.16]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:100001 RT:65000:500001 ET:8 Rmac:fa:59:d4:2a:a8:c5
 *>i[3]:[0]:[32]:[10.0.4.1]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500001 ET:8
Route Distinguisher: 10.0.4.1:6
 *>i[5]:[0]:[24]:[10.0.20.0]
                    10.0.4.1(proxmox1-1)
                                             0    100      0 ?
                    RT:65000:100001 ET:8 Rmac:fa:59:d4:2a:a8:c5
Route Distinguisher: 10.0.4.1:7
 *>i[3]:[0]:[32]:[10.0.4.1]
                    10.0.4.1(proxmox1-1)
                                                  100      0 i
                    RT:65000:500002 ET:8
Route Distinguisher: 10.0.4.3:2
 *>i[5]:[0]:[0]:[0.0.0.0]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100002 ET:8 Rmac:f6:36:76:14:44:a1
 *>i[5]:[0]:[0]:[::] 10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100002 ET:8 Rmac:f6:36:76:14:44:a1
Route Distinguisher: 10.0.4.3:3
 *>i[5]:[0]:[0]:[0.0.0.0]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100003 ET:8 Rmac:a2:dc:16:c1:6f:cd
 *>i[5]:[0]:[0]:[::] 10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100003 ET:8 Rmac:a2:dc:16:c1:6f:cd
Route Distinguisher: 10.0.4.3:4
 *>i[3]:[0]:[32]:[10.0.4.3]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500003 ET:8
Route Distinguisher: 10.0.4.3:5
 *>i[2]:[0]:[48]:[32:62:d4:b9:d6:e3]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[52:05:b6:45:ad:4d]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[52:05:b6:45:ad:4d]:[32]:[10.0.20.19]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100001 RT:65000:500001 ET:8 Rmac:96:d6:ef:8f:d0:a4
 *>i[2]:[0]:[48]:[ca:d2:d3:a0:b3:ce]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[2]:[0]:[48]:[d6:9f:26:a2:04:40]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
 *>i[3]:[0]:[32]:[10.0.4.3]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500001 ET:8
Route Distinguisher: 10.0.4.3:6
 *>i[5]:[0]:[24]:[10.0.20.0]
                    10.0.4.3(proxmox1-3)
                                             0    100      0 ?
                    RT:65000:100001 ET:8 Rmac:96:d6:ef:8f:d0:a4
Route Distinguisher: 10.0.4.3:7
 *>i[2]:[0]:[48]:[bc:24:11:f5:32:01]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500002 ET:8 MM:1
 *>i[2]:[0]:[48]:[bc:24:11:f5:32:01]:[32]:[10.0.21.2]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:100002 RT:65000:500002 ET:8 MM:1 Rmac:f6:36:76:14:44:a1
 *>i[3]:[0]:[32]:[10.0.4.3]
                    10.0.4.3(proxmox1-3)
                                                  100      0 i
                    RT:65000:500002 ET:8
Route Distinguisher: 10.0.4.4:2
 *> [5]:[0]:[0]:[0.0.0.0]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [5]:[0]:[0]:[::] 10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100001 Rmac:52:51:41:96:b3:70
Route Distinguisher: 10.0.4.4:3
 *> [5]:[0]:[0]:[0.0.0.0]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100002 Rmac:f6:df:8f:9c:a0:16
 *> [5]:[0]:[0]:[::] 10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100002 Rmac:f6:df:8f:9c:a0:16
Route Distinguisher: 10.0.4.4:4
 *> [5]:[0]:[0]:[0.0.0.0]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100003 Rmac:9a:a6:c1:68:2b:43
 *> [5]:[0]:[0]:[::] 10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:100003 Rmac:9a:a6:c1:68:2b:43
Route Distinguisher: 10.0.4.4:5
 *> [3]:[0]:[32]:[10.0.4.4]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500003
Route Distinguisher: 10.0.4.4:6
 *> [2]:[0]:[48]:[12:58:ed:f9:9f:c0]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
 *> [2]:[0]:[48]:[12:58:ed:f9:9f:c0]:[32]:[10.0.20.23]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [2]:[0]:[48]:[52:5a:13:a8:73:81]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
 *> [2]:[0]:[48]:[52:5a:13:a8:73:81]:[32]:[10.0.20.32]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [2]:[0]:[48]:[9e:c1:fa:5a:f3:07]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
 *> [2]:[0]:[48]:[9e:c1:fa:5a:f3:07]:[32]:[10.0.20.10]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [2]:[0]:[48]:[de:d6:ab:7b:95:f1]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
 *> [2]:[0]:[48]:[de:d6:ab:7b:95:f1]:[32]:[10.0.20.13]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001 RT:65000:100001 Rmac:52:51:41:96:b3:70
 *> [3]:[0]:[32]:[10.0.4.4]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500001
Route Distinguisher: 10.0.4.4:7
 *> [3]:[0]:[32]:[10.0.4.4]
                    10.0.4.4(proxmox1-4)
                                                       32768 i
                    ET:8 RT:65000:500002


Displayed 48 out of 48 total prefixes

/etc/frr/frr.conf of node 10.0.4.4:

Code:

frr version 8.5.1
frr defaults datacenter
hostname proxmox1-4
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_z100001
 vni 100001
 ip route 10.0.21.0/24 null0
 ip route 10.0.24.0/24 null0
exit-vrf
!
vrf vrf_z100002
 vni 100002
 ip route 10.0.20.0/24 null0
 ip route 10.0.24.0/24 null0
exit-vrf
!
vrf vrf_z100003
 vni 100003
 ip route 10.0.20.0/24 null0
 ip route 10.0.21.0/24 null0
exit-vrf
!
router bgp 65000
 bgp router-id 10.0.4.4
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 10.0.4.1 peer-group VTEP
 neighbor 10.0.4.3 peer-group VTEP
 !
 address-family ipv4 unicast
  import vrf vrf_z100001
  import vrf vrf_z100002
  import vrf vrf_z100003
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_z100001
  import vrf vrf_z100002
  import vrf vrf_z100003
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_z100001
 bgp router-id 10.0.4.4
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
router bgp 65000 vrf vrf_z100002
 bgp router-id 10.0.4.4
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
router bgp 65000 vrf vrf_z100003
 bgp router-id 10.0.4.4
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match evpn vni 100002
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN deny 2
 match evpn vni 100001
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 3
exit
!
route-map MAP_VTEP_OUT permit 1
 match evpn vni 100003
 match evpn route-type prefix
 set metric 200
exit
!
route-map MAP_VTEP_OUT permit 2
exit
!
line vty

kemeris · Jan 7, 2024

/etc/frr/frr.conf of node 10.0.4.3:

Code:

frr version 8.5.1

frr defaults datacenter

hostname proxmox1-3

log syslog informational

service integrated-vtysh-config

!

!

vrf vrf_z100001

 vni 100001

exit-vrf

!

vrf vrf_z100002

 vni 100002

 ip route 10.0.20.0/24 null0

 ip route 10.0.24.0/24 null0

exit-vrf

!

vrf vrf_z100003

 vni 100003

 ip route 10.0.20.0/24 null0

 ip route 10.0.21.0/24 null0

exit-vrf

!

router bgp 65000

 bgp router-id 10.0.4.3

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 no bgp default ipv4-unicast

 coalesce-time 1000

 neighbor VTEP peer-group

 neighbor VTEP remote-as 65000

 neighbor VTEP bfd

 neighbor 10.0.4.1 peer-group VTEP

 neighbor 10.0.4.4 peer-group VTEP

 !

 address-family ipv4 unicast

  import vrf vrf_z100002

  import vrf vrf_z100003

 exit-address-family

 !

 address-family ipv6 unicast

  import vrf vrf_z100002

  import vrf vrf_z100003

 exit-address-family

 !

 address-family l2vpn evpn

  neighbor VTEP route-map MAP_VTEP_IN in

  neighbor VTEP route-map MAP_VTEP_OUT out

  neighbor VTEP activate

  advertise-all-vni

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100001

 bgp router-id 10.0.4.3

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  advertise ipv4 unicast

  advertise ipv6 unicast

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100002

 bgp router-id 10.0.4.3

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  default-originate ipv4

  default-originate ipv6

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100003

 bgp router-id 10.0.4.3

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  default-originate ipv4

  default-originate ipv6

 exit-address-family

exit

!

route-map MAP_VTEP_IN permit 1

exit

!

route-map MAP_VTEP_OUT permit 1

 match evpn vni 100003

 match evpn route-type prefix

 set metric 200

exit

!

route-map MAP_VTEP_OUT permit 2

 match evpn vni 100002

 match evpn route-type prefix

 set metric 200

exit

!

route-map MAP_VTEP_OUT permit 3

exit

!

line vty

/etc/frr/frr.conf of node 10.0.4.1:

Code:

frr version 8.5.1

frr defaults datacenter

hostname proxmox1-1

log syslog informational

service integrated-vtysh-config

!

!

vrf vrf_z100001

 vni 100001

exit-vrf

!

vrf vrf_z100002

 vni 100002

 ip route 10.0.20.0/24 null0

 ip route 10.0.24.0/24 null0

exit-vrf

!

vrf vrf_z100003

 vni 100003

 ip route 10.0.20.0/24 null0

 ip route 10.0.21.0/24 null0

exit-vrf

!

router bgp 65000

 bgp router-id 10.0.4.1

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 no bgp default ipv4-unicast

 coalesce-time 1000

 neighbor VTEP peer-group

 neighbor VTEP remote-as 65000

 neighbor VTEP bfd

 neighbor 10.0.4.3 peer-group VTEP

 neighbor 10.0.4.4 peer-group VTEP

 !

 address-family ipv4 unicast

  import vrf vrf_z100002

  import vrf vrf_z100003

 exit-address-family

 !

 address-family ipv6 unicast

  import vrf vrf_z100002

  import vrf vrf_z100003

 exit-address-family

 !

 address-family l2vpn evpn

  neighbor VTEP route-map MAP_VTEP_IN in

  neighbor VTEP route-map MAP_VTEP_OUT out

  neighbor VTEP activate

  advertise-all-vni

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100001

 bgp router-id 10.0.4.1

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  advertise ipv4 unicast

  advertise ipv6 unicast

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100002

 bgp router-id 10.0.4.1

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  default-originate ipv4

  default-originate ipv6

 exit-address-family

exit

!

router bgp 65000 vrf vrf_z100003

 bgp router-id 10.0.4.1

 no bgp hard-administrative-reset

 no bgp graceful-restart notification

 !

 address-family ipv4 unicast

  redistribute connected

 exit-address-family

 !

 address-family ipv6 unicast

  redistribute connected

 exit-address-family

 !

 address-family l2vpn evpn

  default-originate ipv4

  default-originate ipv6

 exit-address-family

exit

!

route-map MAP_VTEP_IN deny 1

 match evpn vni 100003

 match evpn route-type prefix

exit

!

route-map MAP_VTEP_IN permit 2

exit

!

route-map MAP_VTEP_OUT permit 1

 match evpn vni 100002

 match evpn route-type prefix

 set metric 200

exit

!

route-map MAP_VTEP_OUT permit 2

exit

!

line vty

spirit said:
This is the bug with multiple exit-nodes currently, where a filtering of default route is broken.
(I have sent fix last month, but it's not yet available in repos)

If there is nothing I can do, I will wait for bugfix.

spirit · Jan 8, 2024

mmm, this is strange, I still see others others as exit-nodes in their frr.conf

" default-originate ipv4"

(so you are still having the bug with multiple exit-nodes)

Are you sure to have correctly apply/reload the sdn configuration ?

I'll look at the gui for exit-node primary, it shouldn't be mandatory. (but I don't think it's a problem for your bug here)

About "exit-nodes-local-routing", you can remove it, it's only needed if you want to reach the vm ip from the hypervisor ip itself.

kemeris · Jan 8, 2024

I am sorry for my late response.

spirit said:
mmm, this is strange, I still see others others as exit-nodes in their frr.conf

" default-originate ipv4"

(so you are still having the bug with multiple exit-nodes)

Are you sure to have correctly apply/reload the sdn configuration ?

I'll look at the gui for exit-node primary, it shouldn't be mandatory. (but I don't think it's a problem for your bug here)

About "exit-nodes-local-routing", you can remove it, it's only needed if you want to reach the vm ip from the hypervisor ip itself.

Yes, I removed it from one zone only. Now I removed it from all zones and there is no more "default-originate ipv4" in frr.conf and ECMP routes in routing table.

spirit said:
About "exit-nodes-local-routing", you can remove it, it's only needed if you want to reach the vm ip from the hypervisor ip itself.

Yes, I know that from your post in other threads.

I think I have found why I can't access (from router) some VM IP's while can access others within same Zone on same node.

All VM's I can't access had two interfaces with one public IP/gateway attached to node's bridge.
IP=10.0.20.32/24 gateway=none -> EVPN
IP=5.133.66.48/24 gateway=5.133.66.1 -> Node bridge

Once I switched gateway (10.0.20.1) to evpn interface, I can access VM from router. Will I be able to use public IP as VM's default gateway if I will move public IP into separate Vnet subnet?

spirit · Jan 9, 2024

kemeris said:
Once I switched gateway (10.0.20.1) to evpn interface, I can access VM from router. Will I be able to use public IP as VM's default gateway if I will move public IP into separate Vnet subnet?

you can only have 1 default gw in your vm, so it must be on public ip.

if you need to to reach private ip, from another private network, you need to a static route in your vm like:

"ip route add 10.0.0.0/8 gw 10.0.20.1"

kemeris · Jan 11, 2024

spirit said:
you can only have 1 default gw in your vm, so it must be on public ip.

if you need to to reach private ip, from another private network, you need to a static route in your vm like:

"ip route add 10.0.0.0/8 gw 10.0.20.1"

Thank you spirit for your time, I really appreciate that.

Can confirm "ip route add 10.0.0.0/8 via 10.0.20.1" works, I am able to reach private/public IPs within VNet from router.
Had to set

Code:

net.ipv4.conf.default.rp_filter = 0

and

Code:

net.ipv4.conf.all.rp_filter = 0

as your mentioned in this thread.

As I understand it's not possible to route traffic from certain VM's in VNet to PVE node (via private IP)? At the moment I simply attach secondary VM interface to bridge on PVE node to run automation tasks to PVE API.

Search

Search

SDN with EVPN Controller

kemeris

Member

spirit

Distinguished Member

kemeris

Member

spirit

Distinguished Member

kemeris

Member

kemeris

Member

spirit

Distinguished Member

kemeris

Member

spirit

Distinguished Member

kemeris

Member