[SOLVED] SDN with EVPN Controller

[stitch]

Hello,

I have been exploring SDN and wanted to try it on Proxmox. Right now, I have an experimental cluster with two nodes.

The first thing I noticed is that there aren't the well-known SDN controllers like OpenDaylight, Ryu, and the others. And I'm not familiar with EVPN or BGP as controllers, can you please explain their functions as SDN controllers or are they just doing the routing.

Then I tried some examples from the documentation:
- VXLAN working but not needing a controller is confusing.
- EVPN (controller + zone) working as per the example, I tried adding another zone and VMs from the two aren't pinging.

I don't fully understand EVPN yet, but I need to know if this is the right way to deploy SDN in a datacenter.

Thank you for reading.

 

[spirit]

Hi,
currently they are no openflow sdn controller implementation, only bgp-evpn.

The main advantage of evpn, is that your have a distributed controller, not a central controller , so it's matching well with proxmox.

I have some prelimary implementation of faucet controller (which used ryu), but currently It's missing some part in proxmox to registered ip/mac address in the controller, and mainly ip management with vm. (evpn is auto learning mac/ip from bridge).

Simple vxlan tunnel are indeed working without a controller, but their are flat layer2 tunnel.
Evpn allow anycast gateway, and inter vxlan routing (+ outside network acccess), but allow pilot the control plane, so arp is filtering for example, that's great for big network.
(With evpn, the bgp protocol exchange the ip/mac of the vms between hosts, and push them in bridge. Instead of using arp/nd broadcast to listen mac)


Evpn is allow an rfc standard, that's a big advantage vs openflow, as a lot of switchs vendor implemented it (arista, cisco, juniper).

I'm using evpn at work, 4000 vms, 4 distants datacenters, and yes it's working really fine

Note that a "zone", is an isolation of a group vnets, so it's normal that you can't ping between zones.


Maybe can you share your /etc/pve/sdn/*.cfg config ?

I'll help you if needed.

 

[stitch]

Hi again,

Thank you that was very helpful, but I am still struggling with the zone concept (I assume the 4 datacenters aren't in the same evpn zone, because they still need to communicate). And if I'm only simulating one datacenter for now, using one zone managed by a controller and using firewall rules (or permissions still not sure) to manage will be enough? and how can I make internet access possible from inside the zone?

I'd also like to know if the recommended architecture is leaf and spine since I thought I'd use core-aggregation-access.

Here are the .cfg, as already mentioned I tried the VXLAN and EVPN examples from the documentation so it's a mess:
Controllers.cfg
evpn: myEvpn asn 65000 peers 192.168.248.133,192.168.248.134

Subnets.cfg
subnet: myEzone-10.0.1.0-24 vnet EvpnVnet gateway 10.0.1.1 subnet: myEzone-10.0.2.0-24 vnet EvpnVne2 gateway 10.0.2.1 subnet: myE2zone-10.0.3.0-24 vnet EvpnVne3 gateway 10.0.3.1 subnet: myE2zone-10.0.4.0-24 vnet EvpnVne4 gateway 10.0.4.1

Vnets.cfg
vnet: myVnet zone myVZone tag 500 vnet: sdVnet zone sdVZone tag 1000 vnet: EvpnVnet zone myEzone tag 11000 vnet: EvpnVne2 zone myEzone tag 12000 vnet: EvpnVne3 zone myE2zone tag 13000

Zones.cfg
vxlan: myVZone peers 192.168.248.133,192.168.248.134 ipam pve mtu 1450 vxlan: sdVZone peers 192.168.248.133, 192.168.248.134 ipam pve mtu 1450 evpn: myEzone controller myEvpn vrf-vxlan 10000 exitnodes pve,pve2 ipam pve mac CE:CD:95:64:4D:FF mtu 1450 evpn: myE2zone controller myEvpn vrf-vxlan 11000 exitnodes pve,pve2 ipam pve mac 3A:CF:11:58:9C:60 mtu 1450

Thank you for reading!

 

[spirit]

a zone, is simply a security domain and a type of network (vlan, vxlan, evpn ,....)

vnets in differents can't communicate together. (with evpn, you can route between 2 vnets in the same zone through subnet gateway).

(also in the zone, you can restrict the zones where the vnets are pushed)

I'd also like to know if the recommended architecture is leaf and spine since I thought I'd use core-aggregation-access.

Not mandatory. you can use flat layer2/core/aggreation/access if you want. You can also use spine-leaf with layer3 + bgp.
I'm using both in production.



So, about your config, I see that your are doing vxlan zones && evpn zones, both a differents.
vxlan zones are simply flat layer2 vxlan tunnel, no routing (and no external access, no controller)
evpn zones use a controller (the frr routing daemon), and also the difference, is that the gateway defined on the subnet is pushed locally on each vnet on each host (anycast gateway ip).
This should be your gateway in your vm guest config.

. And if I'm only simulating one datacenter for now, using one zone managed by a controller and using firewall rules (or permissions still not sure) to manage will be enough?

yes , just start with 1 evpn zone + 1 controller for now. (firewall is not related here, it's working like classic pve firewall)

and how can I make internet access possible from inside the zone?

you need to define 1 or 2 exit gateways noes in the evpn zone configuration.
the traffic will go out to external network through the default gw of theses nodes. (you can also enable snat on the subnet if needed).


The flow is for example:

vm(10.0.3.10)---->vnet EvpnVne3(10.0.3.1)------>exitnodes(pve/pve2) snat----->(default gw)---------->(192.168.248.1 )external router (snat)--->internet

if you can try enable snat on the subnet , it's done at exitnode level, but as you have private ip on the exit-nodes, it'll be natted to 192.168.248.133. So maybe you'll have double-nat if you also have nat on your external router.



if you don't enable nat, you'll need to a route in the reverse direction in your external router
something like:

internet -->external router----->route add 10.0.3.0/24 gw 192.168.248.133 gw 192.168.248.134--->exitnodes(pve/pve2)--->vnet EvpnVne3(10.0.3.1)---->vm(10.0.3.10)

(Also, use proxmox 7.2, because some bugfix has been done on multiple exit nodes support last month)


Hope it's helping.

 

[stitch]

Hi sorry for taking too long,

Thank you it's all very helpful. Count this thread as solved I just have some other questions:
- Two nodes linked by a switch, in an EVPN zone with an EVPN controller (being the BGP router). Is BGP working between the two or do I need a bigger network with a at least two routers for there to be a BGP routing?

- I tried simulating a datacenter (node 1) and its backup datacenter (node 2), I created VMs with identical IP addresses (same VNets) on each node to see how the anycast gateway would work and pings weren't successful. Maybe I didn't understand how it's meant to be used.

- About this, does using an EVPN zone imply the use of a VXLAN zone or a VXLAN tunnel?

Evpn allow anycast gateway, and inter vxlan routing (+ outside network acccess), but allow pilot the control plane, so arp is filtering for example, that's great for big network.

- Lastly, do you suggest I try anything else using the EVPN controller or the BGP one (couldn't figure how it's used yet) before I'm done experimenting.

Thank you for reading!

 

[spirit]

- Two nodes linked by a switch, in an EVPN zone with an EVPN controller (being the BGP router). Is BGP working between the two or do I need a bigger network with a at least two routers for there to be a BGP routing?

you can use evpn with only 2 nodes, no problem.

- I tried simulating a datacenter (node 1) and its backup datacenter (node 2), I created VMs with identical IP addresses (same VNets) on each node to see how the anycast gateway would work and pings weren't successful. Maybe I didn't understand how it's meant to be used.

you can't have same ip in 2 vms of course. The anycast ip, is the gateway of the vnet. (on each host, same vnet will have same ip/mac, the gateway of the vm).

About this, does using an EVPN zone imply the use of a VXLAN zone

no, 2 differents zone model

or a VXLAN tunnel?

yes. you can see evpn like: vxlan tunnels + controllers. (main difference: you can route between vnet or to external, arp broadcast are filtered).

- Lastly, do you suggest I try anything else using the EVPN controller or the BGP one (couldn't figure how it's used yet) before I'm done experimenting.

Only evpn controller is used as a zone controller.
The BGP controller, is to override bgp setup on specific host. (so don't use it ).

 

[stitch]

Thank you so much for your assistance and for being patient with me, I really appreciate your dedication to the platform.