SDN evpn clarification

[bublepaw]

Hi,

I am trying to design 2 Proxmox clusters (5 nodes each) each cluster siting in separate geographical location. I want to share only networks for VM's between those sites/cluster. After some reading I came with idea to use SDN with evpn but I do have some questions:

General evpn clarifications

- in my design I want to have two evpn zones in each cluster - my asumption is that I can use single evpn controller for both zones in each cluster - is it correct assumptions or do I need separate evpn controller for each evpn zone
- is there way to split VXLAN and control traffic (bgp) onto separate interfaces
- if I don't specify gateway for specific vnet I am assuming this vent is isolated from rest of vnets in same zone and hosts outside of Proxmox
- is there a good reason to use router reflectors or full mesh is ok - personally I don't see reason to use route reflectors but maybe I am missing something

Routing traffic outside Proxmox cluster

- scenario 1 - I don't use any BGP peering - I am assuming I have to specify exit nodes in Proxmox and on my router to which Proxmox host are pointing using default gateway I need to configure static routes for every subnet that is handled by vnet zones - did I miss something

- scenario 2 - I am peering Proxmox hosts with BGP router using eBGP without evpn support - in that case I am assuming I have to configure bgp controller on every Promox host poiting to my router. Do I have to specify exit nodes if yes should I only configure bgp controller on those hosts or maybe I just configure bgp peering everywhere and don't specify exit nodes but I need than to distribute default route from BGP router

- scenario 3 - I am peering Proxmox hosts with BGP router that supports evpn - where do I configure such peering - also using BGP controller or using peers under evpn controller. In second case should I mark some nodes as exits or I don't need to.

Sharing vnets between clusters (assuming I have same vnets, zones configure with same parameters on both clusters)

- scenario 1 - I am able to reach directly each Proxmox hosts in both clusters - in such case I add all hosts from both clusters into list of peers under evpn controller and I don't need to specify anything else

- scenario 2 - host in both clusters have to talk through default gateway - where do I configure that hosts from other cluster are mutlihop bgp peers. Will such scenario even work - I am assuming yes since there is layer 3 connectivity between all hosts.

- scenario 4 - host in both clusters are peering with bgp routers that supports evpn and have already evpn route from other cluster - should I use route import option in such case to import routes from bgp router or there are some other options I need to configure.

- is there any way to protect that VM's runing on Proxmox cluster in site A not to use nodes in proxmox cluster in site B to send traffic outside evpn zone

- is there any special case where I would need to configure different ASN number in evpn controllers in each cluster other than overlapping node ips

Pawel

 

[jsterr]

What storage are you going to use with that geolocation-splitted systems? Can you make sure the latency is very low so you wont get problems with corosync? What latency (ms) (lowest) is possible?

 

[bublepaw]

What storage are you going to use with that geolocation-splitted systems? Can you make sure the latency is very low so you wont get problems with corosync? What latency (ms) (lowest) is possible?

These are two separate Proxmox clusters each siting in one location which are only sharing networks across WAN - so no problem with storage and corosync

 

[spirit]

Hi,

I am trying to design 2 Proxmox clusters (5 nodes each) siting in separate geographical location sharing same networks for VM's. After some reading I came with idea to use SDN with evpn but I do have some questions:

- in my design I want to have two evpn zones in each cluster - my asumption is that I can use single evpn controller for both zones in each cluster - is it correct assumptions or do I need separate evpn controller for each evpn zone

The more simple it to reuse the use same controller configuration. (the same ASN)
The controller itself (the frr router), is running distributed on each node, and each nodes peer with other nodes.

but, here it seem that you have 2 differents ASN, with routers,... check my response after

- I am using router which supports bgp with evpn (vyos) which sits between those two locations - what are advantages and disadvantages of peering promox hosts with standard bgp or with evpn bgp with this router to connect both site

- if I am peering using standard bgp (no evpn support) I have to add bgp controller for each promox node poiting to my router or should I configure it only for exit nodes. Do I even have to specify exit nodes in case I am peering with bgp on every node

this will need exit-node, and peer exit node with classic bgp.
Personally, if you vyos router already support evpn, simply use them as exit-node. (they just need to announce default 0.0.0.0 to evpn network)

- if I am peering using bgp evpn with my router which is using different ASN should I use bgp controller or just add my router as additional peer in evpn controller (in such case were do I specify ASN number of my router). I am assuming that when I am peering with bgp evpn router I don't need to specify exit nodes

- if I want to share same networks between clusters I need to peer between nodes in both clusters- can I do it through router which sits between locations - in such case where do I specify that remote nodes are multihop peers. Or do I have to use specific network which allows to peer nodes directly bypassing any router between them. Or maybe it is better to use router reflector and peer all nodes in each site to router reflector and then just peer route reflectors between sites
- is there any way to protect that VM's runing on Proxmox cluster in site A not to use nodes in proxmox cluster in site B to send traffic outside evpn zone

I'm running this kind of setup of production.
Multiple datacenter with differents ASN ,

my physical router (arista routers) are doing evpn, and are exit-nodes of each datacenter (and I'm also using them as route reflector on each site).

Each proxmox cluster evpn controller use a different ASN too, and peer with arista route reflectors on each site.

Each arista exit-nodes are exchanging routes evpn with other datacenter arista, and redistribute evpn routers them on each site. (so they are the next-hop if you want to reach a evpn mac/ip on the other datacenter).

Also, you need to configure each proxmox datacenter zone to accept import of remote routes from other proxmox clusters asn.
in the zone configuration, "route target import", you need to configure somethig like: "route-target-import: <remoteevpnasn>:<remotevrfzonevxlan"

 

[bublepaw]

The more simple it to reuse the use same controller configuration. (the same ASN)
The controller itself (the frr router), is running distributed on each node, and each nodes peer with other nodes.

but, here it seem that you have 2 differents ASN, with routers,... check my response after




this will need exit-node, and peer exit node with classic bgp.
Personally, if you vyos router already support evpn, simply use them as exit-node. (they just need to announce default 0.0.0.0 to evpn network)





I'm running this kind of setup of production.
Multiple datacenter with differents ASN ,

my physical router (arista routers) are doing evpn, and are exit-nodes of each datacenter (and I'm also using them as route reflector on each site).

Each proxmox cluster evpn controller use a different ASN too, and peer with arista route reflectors on each site.

Each arista exit-nodes are exchanging routes evpn with other datacenter arista, and redistribute evpn routers them on each site. (so they are the next-hop if you want to reach a evpn mac/ip on the other datacenter).

Also, you need to configure each proxmox datacenter zone to accept import of remote routes from other proxmox clusters asn.
in the zone configuration, "route target import", you need to configure somethig like: "route-target-import: <remoteevpnasn>:<remotevrfzonevxlan"

Thanks - that clarifies a lot. I think I will go with evpn bgp setup although I may have some questions later on spirit. Btw it is amazing work You did with evpn thing

 

[bublepaw]

When I want to peer Proxmox with router which has different ASN I don't see how to do it from UI. If I configure BGP controller it creates entry in frr config which only accepts ipv4 unicast routes unless there is some additional logic which I am not aware which adds evpn routes. On the other hand if I specify router as peer for evpn controller it assumes same ASN number as evpn controller so unless there is hidden option somewhere I will need to add some config entries to frr using templating mechanism - is that correct assumption?

 

[spirit]

When I want to peer Proxmox with router which has different ASN I don't see how to do it from UI. If I configure BGP controller it creates entry in frr config which only accepts ipv4 unicast routes unless there is some additional logic which I am not aware which adds evpn routes. On the other hand if I specify router as peer for evpn controller it assumes same ASN number as evpn controller so unless there is hidden option somewhere I will need to add some config entries to frr using templating mechanism - is that correct assumption?

if you want to peer the evpn overlay network, you need to define the remote peer in the evpn controller. (and it need to use the same evpn asn).

if you want to peer the bgp underlay (for importing classic bgp route for example, or if you use a proxmox node as exit node), you need to add an extra bgp controller and add the remote peer ip and check ebgp option.


in my production setup,

- I'm using bgp for underlay network (full layer3 with point to point interfaces) where each proxmox node use a bgp controller with a different ASN + ebgp , peering with my arista routers with a different ASN too.

- for the evpn overlay network, I'm using same evpn ASN everywhere (in proxmox evpn controller && arista evpn setup)

my arista routers announce default 0.0.0.0 in the evpn network, and is the exit-node, routing to outside non-vxlan subnets