Hopefully this is useful. I believe I can submit a patch if this is acceptable.
Background:
I have a pfSense firewall that separates my current network environment. I wanted to deploy multiple new segments for some workloads I wanted to run. I didn't want to have to keep creating interfaces and vlans for these experiments.
I wanted to use SDN networking to create an isolated environment for my new workloads.
I added a new interface and vlan to pfSense and underlying Proxmox nodes to act as a transit vlan.
I configured a new EVPN zone, added a couple of exit nodes, with exit node local routing enabled and configured with EBGP between proxmox and pfSense.
Problem:
I might not have understood how this was supposed to work. However, I thought that all traffic from my EVPN zone with communicate with the outside world via the transit vlan. I might have misconfigured the zone or other settings. I ended up with asymmetric routing and traffic flowing via a different interface than the transit vlan.
Solution:
I patched three files:
The last change I made was to add the vmbrX interface for the transit vlan into my vrf by manually adding vrf vrf_<vrfname) to the /etc/network/interfaces file and reloading.
Result:
This has resulted in the following output which is what I intended at the start:
Normal proxmox node routing table with no vrf entries in it:
A vrf routing table with their respective entries:
The same commands from inside vtysh:
With the BGP peer configuration automatically built for you on the exit node in frr:
Background:
I have a pfSense firewall that separates my current network environment. I wanted to deploy multiple new segments for some workloads I wanted to run. I didn't want to have to keep creating interfaces and vlans for these experiments.
I wanted to use SDN networking to create an isolated environment for my new workloads.
I added a new interface and vlan to pfSense and underlying Proxmox nodes to act as a transit vlan.
I configured a new EVPN zone, added a couple of exit nodes, with exit node local routing enabled and configured with EBGP between proxmox and pfSense.
Problem:
I might not have understood how this was supposed to work. However, I thought that all traffic from my EVPN zone with communicate with the outside world via the transit vlan. I might have misconfigured the zone or other settings. I ended up with asymmetric routing and traffic flowing via a different interface than the transit vlan.
Solution:
I patched three files:
- /usr/share/perl5/PVE/Network/SDN/Zones/BgpPlugin.pm
- /usr/share/perl5/PVE/Network/SDN/Zones/EvpnPlugin.pm
- /usr/share/pve-manager/js/pvemanagerlib.js
1. I added a vrf_zone property to the peer configuration in BgpPlugin.pm
2. Changed EvpnPlugin.pm to use the vrf_zone property to define the BGP session inside the vrf
3. Added an extra field to the web interface for the BGP controller option to select a valid vrf from the dropdown box
The last change I made was to add the vmbrX interface for the transit vlan into my vrf by manually adding vrf vrf_<vrfname) to the /etc/network/interfaces file and reloading.
Result:
This has resulted in the following output which is what I intended at the start:
Normal proxmox node routing table with no vrf entries in it:
Bash:
root@prox1:/# ip route
default via 192.168.132.1 dev vmbr0 proto kernel onlink
10.10.20.0/24 dev vmbr20 proto kernel scope link src 10.10.20.7
10.10.21.0/24 dev vmbr21 proto kernel scope link src 10.10.21.7
192.168.132.0/24 dev vmbr0 proto kernel scope link src 192.168.132.7A vrf routing table with their respective entries:
Bash:
root@prox1:/# ip route show vrf vrf_public
default nhid 38 via 192.168.254.1 dev vmbr14 proto bgp metric 20
192.168.222.0/24 dev dmz proto kernel scope link src 192.168.222.1
192.168.254.0/24 dev vmbr14 proto kernel scope link src 192.168.254.7The same commands from inside vtysh:
Bash:
prox1# sh ip route
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
IPv4 unicast VRF default:
K>* 0.0.0.0/0 [0/0] via 192.168.132.1, vmbr0 onlink, weight 1, 14:05:05
C>* 10.10.20.0/24 is directly connected, vmbr20, weight 1, 14:05:05
L>* 10.10.20.7/32 is directly connected, vmbr20, weight 1, 14:05:05
C>* 10.10.21.0/24 is directly connected, vmbr21, weight 1, 14:05:05
L>* 10.10.21.7/32 is directly connected, vmbr21, weight 1, 14:05:05
C>* 192.168.132.0/24 is directly connected, vmbr0, weight 1, 14:05:05
L>* 192.168.132.7/32 is directly connected, vmbr0, weight 1, 14:05:05
prox1# sh ip route vrf vrf_public
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
IPv4 unicast VRF vrf_public:
B>* 0.0.0.0/0 [20/0] via 192.168.254.1, vmbr14, weight 1, 00:01:22
C>* 192.168.222.0/24 is directly connected, dmz, weight 1, 13:16:09
L>* 192.168.222.1/32 is directly connected, dmz, weight 1, 13:16:09
C>* 192.168.254.0/24 is directly connected, vmbr14, weight 1, 00:01:23
L>* 192.168.254.7/32 is directly connected, vmbr14, weight 1, 00:01:23With the BGP peer configuration automatically built for you on the exit node in frr:
Bash:
router bgp 65001 vrf vrf_public
bgp router-id 10.10.21.7
no bgp hard-administrative-reset
no bgp graceful-restart notification
neighbor BGP peer-group
neighbor BGP remote-as external
neighbor BGP bfd
neighbor 192.168.254.1 peer-group BGP
!
address-family ipv4 unicast
redistribute connected
neighbor BGP soft-reconfiguration inbound
exit-address-family
!
address-family ipv6 unicast
redistribute connected
exit-address-family
!
address-family l2vpn evpn
default-originate ipv4
default-originate ipv6
exit-address-family
exit