evpn? network segmentation?

[ze42]

Hello,

I'm trying to figure out what EVPN really is, and how it can help with network segmentation.


What I already managed to do :
* Use VXLAN between my nodes to have VM in dedicaded vnet communicate
* Have gateways in VM to nicely split firewall, and completly independant routing tables
* Proxmox just has bridges, and no required knowledge of the IP within the VM and their networks
* Proxmox CAN know about it, and still have some firewall on the links, much like SecurityGroups applied on interfaces on AWS

What i'm trying to figure is :
* Is it possible to use Proxmox directly as a router, with different routing tables?

Like having multiple zones that have completly separated independant routing tables.
* Using proxmox as router between the networks of each zone, with possibly overlapping networks
* Easy firewall to configure within that zone
* Have one of the zone able to route to an external default gateway, that might not be the same as for the node itself (ie: keeping the node in a dedicated internal zone)

 

[nico1875p]

Hello.

Are you trying to build something similar to an AWS VPC multi-tenant architecture, with a VRF per zone? Or NSX like?

You can build a PoC, but if your goal is true VRF-based routing separation (independent routing tables, overlapping IP ranges), then yes
You will need manual Linux VRF configuration and possibly FRR.

Proxmox SDN alone does not provide native VRF-per-tenant abstraction.

What is missing natively in Proxmox :

• No native VRF-per-zone (no separate routing table per tenant in the GUI)
• No built-in tenant-aware routing domains
• No clean support for overlapping IP ranges without manual VRF setup
• No VRF-aware firewall
• No cloud-style multi-tenant network control-plane (like AWS VPC / NSX VMware / OpenStack)
• Proxmox SDN provides L2/L3 segmentation, but does not provide full VRF-based tenant routing isolation.
  
  

 

[ze42]

Ok, so without native support, what I had in mind at start, from what I see now...

Within a single Proxmox:
* create a namespace for each dedicated "zone"
* create bridges, firewall rules, ... in that namespace
* create veth, one in the "zone" namespace, and one in the default namespace
* In proxmox, create the wanted bridges. Manually add the veth from default namespace in it
* Place the VM on that bridge in default namespace

Proxmox default namespace will just use the bridge feature to let the traffic flow to the "zone" namespace
The "zone" namespace will handle firewalling and routing that will have been manually configured.

Not sure how to scale that to a cluster, but at least I get the idea of what is doable.


What about EVPN ?
Is there any simple example of what it really does, how to set it up, ... to get a better understanding on what it can do on a simple single cluster setup ?

 

[nico1875p]

What about EVPN ?
Is there any simple example of what it really does, how to set it up, ... to get a better understanding on what it can do on a simple single cluster setup ?

Without EVPN:

• A VLAN exists only locally on each node.
• If a VM moves to another node, you must ensure the VLAN exists and is correctly trunked everywhere.
• L2 domains depend entirely on physical switch configuration.

With EVPN (VXLAN + BGP):

• Each node becomes part of an overlay network.
• A VNet (VNI) is distributed across all nodes.
• MAC/IP information is exchanged dynamically via BGP.
• The L2 domain is stretched across the cluster using VXLAN

Simple firewall example​

Imagine a 3-node cluster with EVPN enabled.

You create:
- An EVPN zone
- A BGP controller (EVPN control-plane)
- A VNet (e.g., VNI 10050)

Deploy a firewall VM:

• Interface 1 → vmbr0 (WAN / uplink)
• Interface 2 → EVPN VNet (LAN)

Now:

• A VM on node 3 connected to the EVPN VNet
• Can reach the firewall running on node 1
• Even though they are on different physical hosts

Why?
Because:

• The firewall’s MAC/IP is advertised via BGP EVPN
• Traffic is encapsulated using VXLAN between nodes
• Reduced dependency on physical VLAN trunk configuration across switches

You can also add CARP (open-source firewalls) to provide L3 redundancy between two firewall VMs.

For example:

• Firewall 1 on node 1
• Firewall 2 on node 2
• Both connected to the same EVPN VNet (LAN)

If one node crashes:

• EVPN still provides the distributed L2 network across the remaining nodes as long as underlay IP connectivity remains available
  The secondary firewall becomes active via CARP
• The virtual IP remains reachable
• VMs keep connectivity without changing their gateway
  
  
  
  This works very well in practice.
  
  What you described with namespaces is conceptually similar, but limited to a single node. EVPN provides a distributed control-plane (BGP) and scales that concept across all nodes in the cluster.
  

 

[ze42]

Without EVPN:

• A VLAN exists only locally on each node.
• If a VM moves to another node, you must ensure the VLAN exists and is correctly trunked everywhere.
• L2 domains depend entirely on physical switch configuration.

When you say without EVPN, in fact you mean without VXLAN.

VXLAN would be enough to have L2 extend to the whole cluster, EVPN is not required to do that.

With EVPN (VXLAN + BGP):

• Each node becomes part of an overlay network.
• A VNet (VNI) is distributed across all nodes.
• MAC/IP information is exchanged dynamically via BGP.
• The L2 domain is stretched across the cluster using VXLAN

Only part really related to EVPN vs VXLAN is the MAC/IP exchanged via BGP, rather than letting the broadcast do its work and discover them like on a normal switch.

Simple firewall example​

Imagine a 3-node cluster with EVPN enabled.

You create:
- An EVPN zone
- A BGP controller (EVPN control-plane)
- A VNet (e.g., VNI 10050)

Deploy a firewall VM:

• Interface 1 → vmbr0 (WAN / uplink)
• Interface 2 → EVPN VNet (LAN)

Now:

• A VM on node 3 connected to the EVPN VNet
• Can reach the firewall running on node 1
• Even though they are on different physical hosts

Why?
Because:

• The firewall’s MAC/IP is advertised via BGP EVPN
• Traffic is encapsulated using VXLAN between nodes
• Reduced dependency on physical VLAN trunk configuration across switches

You can also add CARP (open-source firewalls) to provide L3 redundancy between two firewall VMs.

For example:

• Firewall 1 on node 1
• Firewall 2 on node 2
• Both connected to the same EVPN VNet (LAN)

If one node crashes:

• EVPN still provides the distributed L2 network across the remaining nodes as long as underlay IP connectivity remains available
  The secondary firewall becomes active via CARP
• The virtual IP remains reachable
• VMs keep connectivity without changing their gateway
  
  
  
  This works very well in practice.
  
  What you described with namespaces is conceptually similar, but limited to a single node. EVPN provides a distributed control-plane (BGP) and scales that concept across all nodes in the cluster.

Pour example would work well with just VXLAN. The hypervisor does not need to know anything about the internal adresses, and EVPN/BGP only helps a little about routing/discovery, rather than being over broadcast.

I really don't see how L2 propagated with VXLAN without EVPN would really differ from the suggested solution.



From further testing...

Each EVPN zones allows to have completly independant routing tables, and still have proxmox run as anycast gateway for each of the subnet. Allowing them to talk to each other, only using the anycast gateway (local proxmox).

Lets say within a zone, we have VM-A1, VM-A2 on subnet A. VM-B1/B2 on subnet 2 ; VM-A1/B1 on node 1, VM-A2/B2 on node 2.
* VM-A1 could connect to VM-B1 directly via node-1 anycast gateway, without traffic going to node 2.
* VM-A2 could connect to VM-B2 directly via node-2 anycast gateway, without traffic going to node 1.
* VM-A1 could connect to VM-A2 directly via VXLAN, without using the gateway.
* VM-A1 could connect to VM-B2 via node-1 anycast gateway which would forward it to node-2

And as those routes are in a dedicated VRF, you could have an other zone with the same network address without any conflict.

Conflict would only start to get out of your local zone... like using an exit-node and need a route back, or an other firewall gateway that will need to be connected to some "outside"...

 

[nico1875p]

Yes — what you describe is basically the key point.

VXLAN already stretches L2.
EVPN becomes interesting when you use it with VRFs and the distributed anycast gateway.

The real gain is that routing stays local on each node, per zone, with clean isolation and overlapping subnet support — especially once you start having multiple zones and external routing requirements.

So it’s less about “L2 vs L2”, and more about distributed L3 at cluster scale.

 

[ze42]

What are the best practice when you have completely separated networks, that need to get "out" of your proxmox cluster ?

From what I understand, exit-nodes user "default" routes from the node itself, not dependent of the zone they come from, which is not what we would want.

Different ideas I have, not sure how they can work...
- In each VRF, inject a default route to a gateway IP (held by a VM) that can get out. Use VMs that can have redundancy, with an HA IP floating between both instances. VM are getting "out" by having an interface in an other zone.
- In each VRF, inject a route to a bridge that holds a veth that bridge to some physical network, letting L2 get out of the zone to some external network. Not sure how to handle proper return path in such case, as source would not properly uses some "exit-node".

Any other ideas how to properly deals with such ?

 

[nico1875p]

You’re right, the exit-node relies on the host’s main routing table, so it’s not ideal if you want clean per-VRF egress.

In practice, if you want something predictable and clean, you have two realistic options:

• Per-VRF firewall/edge VM
  
  Each VRF has its own default route pointing to a firewall/router VM inside that VRF.
  That VM then connects to a transit/external network and handles NAT and policies there.
  If you need HA, use CARP/VRRP between two edge VMs.
  
  
• External edge device (more “datacenter style”)
  
  Export the VRF routes to a real router/firewall (BGP/VRF aware) and let that device handle north-south traffic.
  That keeps Proxmox as a fabric, not as an edge router.

The L2 bridge/veth idea will work in simple labs, but return traffic and isolation quickly become messy once you scale or add multiple tenants.

try to keep egress routing either inside the VRF (edge VM) or push it to a proper external edge.I wouldn’t rely on the host’s main table for multi-tenant traffic.

 

[spirit]

Hello.

Are you trying to build something similar to an AWS VPC multi-tenant architecture, with a VRF per zone? Or NSX like?

You can build a PoC, but if your goal is true VRF-based routing separation (independent routing tables, overlapping IP ranges), then yes
You will need manual Linux VRF configuration and possibly FRR.

Proxmox SDN alone does not provide native VRF-per-tenant abstraction.

What is missing natively in Proxmox :

• No native VRF-per-zone (no separate routing table per tenant in the GUI)
• No built-in tenant-aware routing domains
• No clean support for overlapping IP ranges without manual VRF setup
• No VRF-aware firewall
• No cloud-style multi-tenant network control-plane (like AWS VPC / NSX VMware / OpenStack)
• Proxmox SDN provides L2/L3 segmentation, but does not provide full VRF-based tenant routing isolation.

each zone is a different vrf in evpn with their own routing table.

(only using exit-node is doing a vrf route leak between the zone && default zone. but If you use multiple zones, I'll command to use physical routers as exit evpn nodes)

so yes, you have overlaping ip range, you have firewall at zone level btw, the control plane is disributed through bgp,...

so, you don't seem to understand how evpn is implement in proxmox sdn ? do you have really tested it ?

 

[nico1875p]

Thanks for the clarification.

You’re right, I misunderstood how EVPN is implemented in Proxmox SDN.

I was mainly thinking in terms of VXLAN L2 stretch and didn’t fully account for the per-zone VRF mapping.
That makes things much clearer.

Apologies for the confusion on my side.