SDN EVPN external gateway

[Lehmann]

Hello,

we try to build a PVE Cluster with a EVPN VXLAN SDN.
The configuration for the VM to VM Communication is working, also SNAT with the Subnet Gateway is working.

Now we try to replace the subnet gateway with an external OPNsense, so that the OPNsense can provide all required network services (DHCP, NAT, etc.). The idea is to connect the OPNsense with a VXLAN interface to the PVE Hosts to build a Layer 2 network between the VMs and the Firewall.
Is this possible and has someone tried this, because we are currently not able to communicate between a VM and the Firewall?

 

[spirit]

If you replace the evpn anycast gateway and want to centralize to en opense vm, the best way is to not use evpn, but a simple vxlan zone.

 

[Lehmann]

Hi,
yes the plan is to replace the anycast gateway but the opnsense is no VM, because then I would simply add different interfaces for the different vnets and remove the gateway entry from the subnet.

But in this case the OPNsense is a separate Hardware, exist there any example for connecting external systems in to the vxlan? Or have you an hint where to find the required connection options?

 

[Lehmann]

I was able to connect the OPNsense to a simple VXLAN Zone (not EPVN) with the following settings.

Zone:

vxlan: vxlanz1
        peers 192.168.250.1,192.168.250.2
        ipam pve

VNET:

vnet: vxlan1
        zone vxlanz1
        tag 1000

After this, the last thing to do is to create an Interface assignment with an IP on the new interface in the OPNsense.
And now the OPNsense can be used as default Gateway for the VMs/LXCs.

 

[spirit]

I was able to connect the OPNsense to a simple VXLAN Zone (not EPVN) with the following settings.

Zone:

vxlan: vxlanz1
        peers 192.168.250.1,192.168.250.2
        ipam pve

VNET:

vnet: vxlan1
        zone vxlanz1
        tag 1000

View attachment 62247

After this, the last thing to do is to create an Interface assignment with an IP on the new interface in the OPNsense.
And now the OPNsense can be used as default Gateway for the VMs/LXCs.

thanks for the report ! Good to known that OPNsense vxlan is working fine.

(evpn is not supported yet on bsd)

 

[gunterwa]

I was able to connect the OPNsense to a simple VXLAN Zone (not EPVN) with the following settings.

Zone:

vxlan: vxlanz1
        peers 192.168.250.1,192.168.250.2
        ipam pve

VNET:

vnet: vxlan1
        zone vxlanz1
        tag 1000

View attachment 62247

After this, the last thing to do is to create an Interface assignment with an IP on the new interface in the OPNsense.
And now the OPNsense can be used as default Gateway for the VMs/LXCs.

May I know WHY you need VXLAN to connect your PVE and OPNsense? Is the OPNsense in far end and there is a L3 network between them?

 

[spirit]

May I know WHY you need VXLAN to connect your PVE and OPNsense? Is the OPNsense in far end and there is a L3 network between them?

evpn network is a vxlan network.

if you don't want to to vxlan directly on opnense (on L2 network), as gateway of your vm (on the same L2 subnet),

if you want to do L3, use anycast gateway as gateway of your vms, and defined an exit-node on the zone (1 of your proxmox node), acting as a router between the vxlan network && the real network. Then forward traffic to your opnsense router.
Then use bgp between this exit-node and the opense router to announce the evpn subnet. (or alternatively, use static routes opnsense).


if freebsd could support evpn , you could use opnsense as exit-node directly, it's a limitation of opnsense.
(vyos router for example is supporting eat)

 

[gunterwa]

evpn network is a vxlan network.

if you don't want to to vxlan directly on opnense (on L2 network), as gateway of your vm (on the same L2 subnet),

if you want to do L3, use anycast gateway as gateway of your vms, and defined an exit-node on the zone (1 of your proxmox node), acting as a router between the vxlan network && the real network. Then forward traffic to your opnsense router.
Then use bgp between this exit-node and the opense router to announce the evpn subnet. (or alternatively, use static routes opnsense).


if freebsd could support evpn , you could use opnsense as exit-node directly, it's a limitation of opnsense.
(vyos router for example is supporting eat)

Hi Spirit,

I am new to PVE SDN and had posted some findings (maybe issues) in this forum for discussion.
For BGP EVPN and p2p VXLAN I still prefer putting them on L3 switch (PVE GW) for two reasons:
1. Easy configuring and simple topology
2. Beter performance
For PVE SDN, online doc just inform how to configure, what I want to know is the application scenarios, especially for PVE running in datacenter providing Iaas service. If you have any, please share .).

Thanks!

 

[spirit]

at work, we are using evpn using arista-router as evpn exit-node (on 3 DC datacenter),

then each vm use proxmox anycast gateway. (so I can migrate the vm from any DC, and the traffic is always using the local router/anycast gw).


Now, if you want to use opense/pfsense, they are no support of evpn, the only possibility is l2 vxlan on pfense or l3bgp on pfsense + proxmox as exit-node.

 

[gunterwa]

at work, we are using evpn using arista-router as evpn exit-node (on 3 DC datacenter),

then each vm use proxmox anycast gateway. (so I can migrate the vm from any DC, and the traffic is always using the local router/anycast gw).


Now, if you want to use opense/pfsense, they are no support of evpn, the only possibility is l2 vxlan on pfense or l3bgp on pfsense + proxmox as exit-node.

Got it, thx .").

 

[lp_xanclas]

Hi @spirit I'm kind interesed in the setup you mention (with Arista-router as exit node), because I'm using vyos router for the similar purpose, and my idea is to have an isolated VRF per tenant (and with it's associated EVPN zone in proxmox) that I can then also instantiate a BGP instance to be used by the tenant. To access internet, the vyos as pseudo-interface with a VLAN in the tenant VRF that connects to a pfsense.
However I'm not sure if this is supported by vyos (or if there's any free router that can support this evpn+vrf+bgp), because at this time (I can share the configuration) I'm able to get the routes of the VMs IPs from the proxmox servers, but I'm unable to reach this same VMs using the router as source.

 

[lp_xanclas]

Ok, found the issue. Forgot to announce a default route.
Now the main problem , if I apply the same configuration for a new tenant, I loose the evpn routes on the first tenant, but got the appropriate routes on the second tenant.
I can share the commands I used.