SDN (EVPN) SNAT with two exit nodes not working

[bodleytunes]

#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3

###################################################################
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1


###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1
#

###################################################################
# Magic system request Key
# 0=disable, 1=enable all, >1 bitmask of sysrq functions
# See https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html
# for what other values do
#kernel.sysrq=438

net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0

 

[bodleytunes]

Main LAN facing bridge

auto vmbr0
iface vmbr0
  bridge-ports vni7 eno1
  bridge-vids 1,7
  bridge-vlan-aware yes
  address 10.12.89.10/24
  gateway 10.12.89.1
 # mtu 9000

 

[bodleytunes]

SDN Generated FRR config

Building configuration...

Current configuration:
!
frr version 8.5.1
frr defaults datacenter
hostname supa1
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
vrf vrf_zone0
 vni 10000
exit-vrf
!
router bgp 65010
 bgp router-id 10.12.89.10
 no bgp default ipv4-unicast
 coalesce-time 1000
 bgp bestpath as-path multipath-relax
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 10.12.89.1 peer-group BGP
 neighbor 10.12.89.11 peer-group VTEP
 neighbor 10.12.89.12 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65010 vrf vrf_zone0
 bgp router-id 10.12.89.10
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast
  advertise ipv6 unicast
  route-target import 4200000000:10000
  route-target export 4200000000:10000
 exit-address-family
exit
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
end

 

[bodleytunes]

BTW I can see when pinging from outside LAN to a vnet the traffic comes into the vnet, the vm on the vnet replies but the reply never leaves the host, its like it gets dropped/blackholed. It does sound a bit like something that would happen when reverse path verification is enabled and its dropping unknown replies.

 

[bodleytunes]

Yeah just tried again, the vm can be on any host and as long as only one exit node is selected in zone setup, it will ping out to the network. This can be a remote node to the one that the VM is residing on. However if you select more than one exit node, the ping stops workinging, seems the odd one gets through initially then it just times out. Mac address of the gateway is learned, nothing changes there.

 

[lunam]

Yep, same issue here if I’m not using SNAT. If i’m using SNAT I can also only use one exit node but maybe it is a different underlying issue?

 

[bodleytunes]

I can see vyos is learning the networks with BGP to go next hop via the 3 proxmox nodes so routing should work, in fact it is in one direction, inbound to the vm's the traffic gets through to the vm on the vnet, its just when the vm reples the packet gets dropped on the return path on the proxmox node.

 

[spirit]

I can see vyos is learning the networks with BGP to go next hop via the 3 proxmox nodes so routing should work, in fact it is in one direction, inbound to the vm's the traffic gets through to the vm on the vnet, its just when the vm reples the packet gets dropped on the return path on the proxmox node.

is rp_filter disabled ?

sysctl -w net.ipv4.conf.default.rp_filter=0
sysctl -w net.ipv4.conf.all.rp_filter=0

you really need this to avoir packet dropped if it's going out a exit-node , and come-back to another exit-node.

 

[niwamo]

Had the same issue. Looked into it and found a bug. Fixed it and posted the solution over here: https://forum.proxmox.com/threads/bugfix-for-evpn-sdn-multiple-exit-nodes.137784/

It's as simple as two or three lines in the plugin, and I scripted out the changes if you want a copy/paste solution.