evpn networks no internet

S

swiftay737

New Member
Feb 20, 2026
2
0
1
Hello
In need of a little help understanding this, ive been looking at a few threads etc, ended up couldn't get it to advertise to opnsense. i then reinstalled proxmox and have got this far lol.
I wanted the evpn vnets to get internet through opnsense host so bgp from proxmox to opnsense to learn routes.


Proxmox SDN: Single PVE Node, Adding 2nd and 3rd Node After.
Fabrics:
OSPF - 10.0.23.0/24
Area - 0.0.0.0
Controllers:
EVPN Controller - 65553, SDN Fabric OSPF
BGP Controller - ASN 65553, Peers=10.0.24.1, EBGP Ticked
Zones: EVPN Zone (ProdNet)
VRF-VXLAN Tag=1000
Exit Nodes (Node Name)
Primary Exit Node (Node Name)

I tried originally with local node routing and advertise subnets ticked under the assumption that advertise subnets meant advertise evpn vnets into bgp controller to opnsense but this seems wrong?
Local node routing for routing between evpn vnets.

With local node routing and advertise subnets i have no internet in a vm on the vnet and can ping 1.1.1.1 but not google,com.
With them both unticked i can ping both and get to internet sites from a fedora vm.

Code: 
Proxmox Generated frr.conf: With Local Node Routing And Advertise Subnets Unticked



frr version 10.4.1
frr defaults datacenter
hostname Alicia
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_ProdNet
 vni 1000
exit-vrf
!
router bgp 65553
 bgp router-id 10.0.23.10
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 coalesce-time 1000
 no bgp graceful-restart notification
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65553
 neighbor VTEP bfd
 neighbor VTEP update-source dummy_ospf-fab
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 3
 neighbor 10.0.24.1 peer-group BGP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_ProdNet
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_ProdNet
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65553 vrf vrf_ProdNet
 bgp router-id 10.0.23.10
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
ip prefix-list only_default seq 1 permit 0.0.0.0/0
!
ipv6 prefix-list only_default_v6 seq 1 permit ::/0
!
route-map MAP_VTEP_IN deny 1
 match ip address prefix-list only_default
exit
!
route-map MAP_VTEP_IN deny 2
 match ipv6 address prefix-list only_default_v6
exit
!
route-map MAP_VTEP_IN permit 3
exit
!
route-map MAP_VTEP_OUT permit 1
exit
router ospf
 ospf router-id 10.0.23.10
exit
!
interface dummy_ospf-fab
 ip ospf area 0.0.0.0
 ip ospf passive
exit
!
interface vmbr1.23
 ip ospf area 0.0.0.0
 ip ospf network point-to-point
exit
!
access-list pve_ospf_ospf-fab_ips permit 10.0.23.0/24
!
route-map pve_ospf permit 100
 match ip address pve_ospf_ospf-fab_ips
 set src 10.0.23.10
exit
!
ip protocol ospf route-map pve_ospf
!
!
line vty
!


Code: 
Proxmox Generated frr.conf: With Local Node Routing Ticked And Advertise Subnets Unticked


  GNU nano 8.4                                                                                                                                /etc/frr/frr.conf                                                                                                                                     
frr version 10.4.1
frr defaults datacenter
hostname Alicia
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_ProdNet
 vni 1000
exit-vrf
!
router bgp 65553
 bgp router-id 10.0.23.10
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 coalesce-time 1000
 no bgp graceful-restart notification
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65553
 neighbor VTEP bfd
 neighbor VTEP update-source dummy_ospf-fab
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 3
 neighbor 10.0.24.1 peer-group BGP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65553 vrf vrf_ProdNet
 bgp router-id 10.0.23.10
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
ip prefix-list only_default seq 1 permit 0.0.0.0/0
!
ipv6 prefix-list only_default_v6 seq 1 permit ::/0
!
route-map MAP_VTEP_IN deny 1
 match ip address prefix-list only_default
exit
!
route-map MAP_VTEP_IN deny 2
 match ipv6 address prefix-list only_default_v6
exit
!
route-map MAP_VTEP_IN permit 3
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
ip route 10.10.10.0/24 10.255.255.2 xvrf_ProdNet
router ospf
 ospf router-id 10.0.23.10
exit
!
interface dummy_ospf-fab
 ip ospf area 0.0.0.0
 ip ospf passive
exit
!
interface vmbr1.23
 ip ospf area 0.0.0.0
 ip ospf network point-to-point
exit
!
access-list pve_ospf_ospf-fab_ips permit 10.0.23.0/24
!
route-map pve_ospf permit 100
 match ip address pve_ospf_ospf-fab_ips
 set src 10.0.23.10
exit
!
ip protocol ospf route-map pve_ospf
!
!
line vty
!
Code: 
Proxmox Generated frr.conf: With Local Node Routing Ticked And Advertise Subnets Ticked



frr version 10.4.1
frr defaults datacenter
hostname Alicia
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_ProdNet
 vni 1000
exit-vrf
!
router bgp 65553
 bgp router-id 10.0.23.10
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 coalesce-time 1000
 no bgp graceful-restart notification
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65553
 neighbor VTEP bfd
 neighbor VTEP update-source dummy_ospf-fab
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 3
 neighbor 10.0.24.1 peer-group BGP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65553 vrf vrf_ProdNet
 bgp router-id 10.0.23.10
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
ip prefix-list only_default seq 1 permit 0.0.0.0/0
!
ipv6 prefix-list only_default_v6 seq 1 permit ::/0
!
route-map MAP_VTEP_IN deny 1
 match ip address prefix-list only_default
exit
!
route-map MAP_VTEP_IN deny 2
 match ipv6 address prefix-list only_default_v6
exit
!
route-map MAP_VTEP_IN permit 3
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
ip route 10.10.10.0/24 10.255.255.2 xvrf_ProdNet
router ospf
 ospf router-id 10.0.23.10
exit
!
interface dummy_ospf-fab
 ip ospf area 0.0.0.0
 ip ospf passive
exit
!
interface vmbr1.23
 ip ospf area 0.0.0.0
 ip ospf network point-to-point
exit
!
access-list pve_ospf_ospf-fab_ips permit 10.0.23.0/24
!
route-map pve_ospf permit 100
 match ip address pve_ospf_ospf-fab_ips
 set src 10.0.23.10
exit
!
ip protocol ospf route-map pve_ospf
!
!
line vty
!


opnsense has a vlan 24 (Transit-PVE)
opnsense has firewall rule on transit-pve to any
opnsense has nat outbound set to hybrid and a rule for
Interface: WAN
Source: 10.10.10.0/24
NAT Address: Interface Address

Questions:
why does advertise subnets seem to not change anything?
Why opnsense learns 10.10.10.0/24 Network when advertise subnets and local node routing is unticked but not ticked?
The EVPN controller & the BGP controller, its correct to use the same ASN?

am i misunderstanding something here?

Summary:

With EVPN Zone Local Node Routing & Advertise Subnets Ticked opnsense doesnt learn the routes anymore, The VM on the evpn vnet with snat ticked can ping 1.1.1.1 but not google.com/ go to internet in browser. it can ping opnsense on the transit network and any other device on network. the firewall alows to any from the vnet (10.10.10.0) on the transit network.
opnsense not learning the routes is the issue and the bgp controller just stops giving it routes it seems.
 
Last edited:
advertise-subnets advertises connected routes into bgp if the node is not an exit-node. On your single node (which is also the exit node), ticking advertise-subnets does nothing extra because the exit node already has that behavior. That's why Configs 2 and 3 are identical -- advertise-subnets only has a real effect on nodes that are NOT exit nodes. exit-node-local-routing means we do not redistribute the connected routes and we do not import them into evpn on the exitnode. So routes stay local to the node -- they never enter the BGP table that OPNsense peers with. That's why OPNsense doesn't see your VNet subnets. So your current setup having advertise-subnets and exit-node-local-routing unticked is correct.
It's also correct to use the same ASN for the BGP and EVPN router, that's best practice.

Let me know if this answers all your questions!
 
Ah, I think i understand, so advertise subnets is for non exit nodes to advertise its vnets into the EVPN Controller for the exit nodes to learn routes to vnets and for inter-vnet traffic?

So if it is a exit node it advertises the vnets by default with or without advertise subnets ticked as it is the node doing the routing.
non exit nodes route to the exit nodes.
this is just a test node, and was going to add the evpn etc into my main 3 node cluster if i got it all working right.
so really i should leave advertise subnets as ticked and when if i added another 2 nodes to this cluster the best practice would be to:

A: have 1 node as an exit node.
B: have all nodes as exit nodes


concerning the local node routing

"your current setup having advertise-subnets and exit-node-local-routing unticked is correct."

when you say this do you mean its correct for advertising to opnsense, you mean its correct when using opnsense as the firewall/router? This mean the inter-vnet is done through opnsense and not pve correct?

if proxmox evpn vnets are snat out of proxmox, so there on accessed through the pve host ip and opnsense doesnt learn routes through the bgp controller anymore. how does my router learn the routes to said vnets if the bgp controller no longer sends them? say im on user vlan 50 accessing vnet3 it would travel to opnsense and out?

i should be using static routes for this? using ai it tells me to configure frr thtough cli which i was going to fiddle with but sdn overwrites the frr.conf. i feel ive just done something wrong or im missing something there.


The thought process:

I recently got ospf up and running between opnsense and my l3 switch (Basic L3 Switch). now opnsense learns the l3 vlans and routes correctly.
Now i wanted to do same on the main pve cluster. in future i plan for bgb offsite (router to router) with a vpn to a friend.

I may keep it as it is with local node routing unticked as well as advertise subnets unticked.
I'm more curious on how it all works and gettting it all working.

when they ticked the vnets dont get internet, they can ping 1.1.1.1, 8.8.8.8, 10.0.24.1, 10.0.0.1, but no internet and not able to ping google.com etc.
so the icmp is leaving proxmox with local node routing on but nothing else. the proxmox firewall is off.

Proxmox Firewall:

In SDN there is Vnet Firewall, i'm under the assumption this is used for inter-vnet traffic when i do turn the firewalls on from datacentre all way through vms and vnets correct? or not really like that?

cant really seem to understand why dns isnt working inside the vnets. it works fine on vlans etc and without local routing ticked but then craps out if i do
so from what i understand my DNS is Working but its something im missing in proxmox? i have setup an adguard dlc inside a seperate vnet, which my fedora vm in vnet 1 can ping the ip of adguard in vnet 2 but nothing else really. i was thinking it was something to do with my opnsense but really am stumped aha
appreciate the help!
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back