# Building a Multi-Site SDN with Proxmox, EVPN over BGP, and ZeroTier
## Problem We Were Solving
We wanted to create a unified **Software Defined Network (SDN)** that spans multiple physical locations, each with its own:
- Local network (`LAN`)
- **Proxmox VE** hypervisor node
- **Ubiquiti UDM Pro** as the edge router
The goal was to create a **shared Layer 2 network** across all locations so that virtual machines (VMs) and containers could move between sites seamlessly and retain reachability.
###
Initial LimitationWhile **EVPN over VXLAN** with **FRR and BGP** was our chosen approach, we ran into a roadblock:
**BGP with EVPN requires multicast**, which is **not supported over traditional IPsec VPNs**.
###
SolutionWe added [**ZeroTier**](https://docs.zerotier.com/ubiquiti/) to each **UDM Pro** and **Proxmox node**, creating an encrypted, software-defined Layer 2 overlay that *does* support what we need for:
- BGP peering
- EVPN functionality
- Seamless routing between SDN subnets
## Setup Overview
Each site includes:
- A UDM Pro with ZeroTier client
- A Proxmox VE node with ZeroTier and FRR installed
- Local area network (`LAN`) with a different subnet at each location
- VM networks attached to a shared SDN using EVPN over VXLAN
### IP layout (from the working setup):
| Site | LAN Subnet | ZeroTier IP | SDN Subnet | VXLAN Subnet |
|-------------|------------------|------------------|----------------|------------------|
| Location A | `10.68.0.0/24` | `172.29.16.10` | `10.69.99.0/24`| `10.69.3.0/24` |
| Location B | `10.0.5.0/24` | `172.29.16.20` | `10.69.99.0/24`| `10.69.3.0/24` |
| Location C | `192.168.10.0/24`| `172.29.16.30` | `10.69.99.0/24`| `10.69.3.0/24` |
All nodes are joined to the **same ZeroTier network** (`172.29.16.0/24`) and use that interface for BGP peering.
---
## Proxmox: `/etc/network/interfaces.d/sdn`
```bash
# SDN Bridge and VXLAN
auto vrf_vnxzone
iface vrf_vnxzone
vrf-table auto
post-up ip route del vrf vrf_vnxzone unreachable default metric 4278198272
auto vrfbr_vnxzone
iface vrfbr_vnxzone
bridge-ports vrfvx_vnxzone
bridge_stp off
bridge_fd 0
mtu 1450
vrf vrf_vnxzone
auto vrfvx_vnxzone
iface vrfvx_vnxzone
vxlan-id 20000
vxlan-local-tunnelip 172.29.16.30
bridge-learning off
bridge-arp-nd-suppress on
mtu 1450
auto vxlan_vxnet0
iface vxlan_vxnet0
vxlan-id 25000
vxlan-local-tunnelip 172.29.16.30
bridge-learning off
bridge-arp-nd-suppress on
mtu 1450
auto vxnet0
iface vxnet0
address 10.69.99.1/24
hwaddress BC:24:11:4E:00:5D
bridge_ports vxlan_vxnet0
bridge_stp off
bridge_fd 0
mtu 1450
ip-forward on
arp-accept on
vrf vrf_vnxzone
```
## Proxmox: `/etc/frr/frr.conf`
```bash
frr version 8.5.2
frr defaults datacenter
hostname proxmox
log syslog informational
service integrated-vtysh-config
vrf vrf_vnxzone
vni 20000
exit-vrf
router bgp 65000
bgp router-id 172.29.16.30
no bgp default ipv4-unicast
coalesce-time 1000
no bgp graceful-restart notification
neighbor VTEP peer-group
neighbor VTEP remote-as 65000
neighbor VTEP bfd
neighbor 172.29.16.1 peer-group VTEP
neighbor 172.29.16.2 peer-group VTEP
neighbor 172.29.16.3 peer-group VTEP
neighbor 172.29.16.10 peer-group VTEP
neighbor 172.29.16.20 peer-group VTEP
neighbor 172.29.16.30 peer-group VTEP
address-family ipv4 unicast
import vrf vrf_vnxzone
exit-address-family
address-family ipv6 unicast
import vrf vrf_vnxzone
exit-address-family
address-family l2vpn evpn
neighbor VTEP activate
neighbor VTEP route-map MAP_VTEP_IN in
neighbor VTEP route-map MAP_VTEP_OUT out
advertise-all-vni
exit-address-family
exit
router bgp 65000 vrf vrf_vnxzone
bgp router-id 172.29.16.30
no bgp graceful-restart notification
address-family ipv4 unicast
redistribute connected
exit-address-family
address-family ipv6 unicast
redistribute connected
exit-address-family
address-family l2vpn evpn
default-originate ipv4
default-originate ipv6
exit-address-family
exit
ip prefix-list only_default seq 1 permit 0.0.0.0/0
ipv6 prefix-list only_default_v6 seq 1 permit ::/0
route-map MAP_VTEP_IN deny 1
match ip address prefix-list only_default
exit
route-map MAP_VTEP_IN deny 2
match ipv6 address prefix-list only_default_v6
exit
route-map MAP_VTEP_IN permit 3
exit
route-map MAP_VTEP_OUT permit 1
exit
```
---
## ️ UDM Pro BGP Configuration (example for Site A)
```bash
router bgp 65000
bgp router-id 172.29.16.1
no bgp default ipv4-unicast
coalesce-time 1000
neighbor VTEP peer-group
neighbor VTEP remote-as 65000
neighbor VTEP bfd
neighbor 172.29.16.2 peer-group VTEP
neighbor 172.29.16.3 peer-group VTEP
neighbor 172.29.16.10 peer-group VTEP
neighbor 172.29.16.20 peer-group VTEP
neighbor 172.29.16.30 peer-group VTEP
address-family l2vpn evpn
neighbor VTEP activate
advertise-all-vni
exit-address-family
```
---
## What We Learned
- **ZeroTier** solved the multicast problem by providing unicast mesh between locations.
- **FRR + EVPN** provided dynamic, stretched L2/L3 connectivity.
- **UDM Pro** could run BGP and dynamically learn EVPN routes from the Proxmox nodes.
##
Final Result- VMs retain reachability when migrated across sites.
- No static routes required on UDMs.
- New locations can join the network by just:
- Installing ZeroTier
- Joining the ZeroTier network
- Adding a BGP config block
## Tools Used
- Proxmox VE
- FRR (Free Range Routing)
- ZeroTier
- Ubiquiti UDM Pro
---
## Reference
- [ZeroTier Docs for Ubiquiti](https://docs.zerotier.com/ubiquiti/)
Last edited: