[SOLVED] Inter node SDN networking using EVPN-VXLAN

[nicof2000]

I also got it working with one BGP controller. But even with this "simpel setup" I experience some problems, where the same config works on one pve node, but not on the other... This is a bit strange, but I'll try to investivate this one on my own. (Edit: This is related to having multiple BGP controller)

In case of maintainence work (when the pve node with the bgp controller will be shut down), I'd like to add two more bgp controller, for the other hosts (ecmp).

As soon as I do this, my test containers become unreachable, if the traffic is being routed over the wrong pve node. To force this behavior I disabled the bgp sessions to the primary exit node. I'm not quiet sure how to debug this properly. Has this been tested at all? Is the fundamental idea of how to setup this correct?

 

[niel]

you can use same ASN for both evpn && bgp


put all your exit-nodes


indeed you don't need static route if you use bgp.



The route is announced by all exit-nodes (even if the vm is not on this node, even if you define exit-node on each).

So, in all case, you'll have ecmp. If a packet in coming to a exit-node when the vm is not present, it'll be rerouted again inside the evpn.


The only way to have direct routing from your fortigate to the evpn, is to have evpn support inside your fortigate (Then the fortigate is the evpn exit-node)




with 3 nodes, it seem to be overkill. just keep fullmesh peering with all nodes ip as evpn peer.

Thank you @spirit

Change of PLANS - I manged to get my self a six port Lanner Device and I installed VYOS 1.4 LTS on it.





I will replace my L2 switch with this VYOS device now. I will connect 1 port to my home router and other to the 3 proxmox nodes


Can you please guide how to setup EVPN-VXLAN on Vyos, you can give me high level steps that I need to follow, I'll try to figure out the details.

I am attaching my current SDN config.





I have removed my node names , but if understand correctly in Exit Nodes I will need to select ALL nodes + VYOS

And primary exit node will be VYOS



I have 5 subnets so I created 5 Vnets, so I can choose the bridge in the VM and it will get the ip address from the subnet

 

[niel]

I have a small 3 node cluster with vmbr0 allocated 192.168.1.0/24 host node IPs. From that LAN network I can reach all internal VM/CTs on each of the 3 host nodes. Fine. Now I have 2 OpenWrt CTs on two different nodes with a pair of Debian CTs also on each of those two nodes. I added a "blank" vmbr1 bridge to each host node where OpenWrt uses vmbr0 for each soft router and gets a 192.168.1.* IP for its wan and vmbr1 for lan (either 192.168.2.1/24 or 192.168.3.1/24). On each respective Debian CT I can ping 1.1.1.1, 192.168.1.* and also the other 192.168.2(or3).* Debian guest CTs "behind" each respective OpenWrt router. All good and working as expected.

Now, using SDN magic, how do I make it so that the Debian CTs behind each of the OpenWrt routers can see the other private network guests behind the other router on the other host node WITHOUT using VPN/wireguard or static routing?

Apologies @markc I may have hijacked your thread... but I have a similar setup and I wanted to get rid of the primary exit node bottleneck

 

[nicof2000]

I haven't try to setup EVPN on VyOS 1.4 yet. I was trying to use regular BGP with VyOS 1.3, so I can't say much about this, but this might help: https://docs.vyos.io/en/sagitta/configexamples/autotest/L3VPN_EVPN/L3VPN_EVPN.html

 

[niel]

I haven't try to setup EVPN on VyOS 1.4 yet. I was trying to use regular BGP with VyOS 1.3, so I can't say much about this, but this might help: https://docs.vyos.io/en/sagitta/configexamples/autotest/L3VPN_EVPN/L3VPN_EVPN.html

Thanks @nicof2000 , This example is between 3 sites I guess... if you have Vyos i think EVPN is the best, just select the exit node as the Vyos router

 

[nicof2000]

Yea, unfortunately I tried to migrate a pve node with a mikrotik router in front of it and this totally failed...

First of all I had problems exporting the routes using bgp (which worked in my lab setup), first it all worked as I expected, then the incoming packages were not routed into the vrf correctly (just 1 host, not in a evpn cluster, but still a evpn zone, to extend it afterwards). After solving this using static route I had the same issue a few hours later, which resulted in the whole network becoming unreachable. For now I have a virtualized router, which acts as a "proxy" to serve the layer 2 networks to this host. I need to debug this in the following days.

 

[niel]

Hi all, I struggled a lot with this and ended up installing ProxMox on the Lanner device and added it to my current cluster, I made the lanner a part of the evpn controller and selected it as the as an exit node in the zone and everything is working great... no need for any additional configuration

 

[markc]

Hi niel, would you mind elaborating on your solution and which Lanner device?

 

[niel]

Hi niel, would you mind elaborating on your solution and which Lanner device?

Hi Markc, if your proxmox nodes are gigabit you may use https://www.ebay.com.au/itm/126559885247

If you proxmox nodes are 10gigabit you may use https://teklager.se/en/products/routers/tlsense-10gbps-intel-atom-c3758r

As I mentioned in my last post, I installed ProxMox on it like a normal mini pc and then, I put all the ports in the same vmr0 bridge, so now the new device will act like a switch with one ip address.

Connect your 3 nodes to device and 1 port to your router.

for eg node1,node2,node3, node 4 = 192.168.10,192.168.11,192.168.12,192.168.13,

Add it to your existing cluster and make it a part of your SDN, and mark it as the primary exit node.

Than on your external router make a summary route of all your internal ip address ranges to the ip address of the new device, thats it.