SDN (EVPN) communication between proxmox clusters

[nikordev]

Hi all!

I have proxmox cluster with vswitch vlan (from provider) between nodes and EVPN network between VMs, everything work fine

Now i need a new cluster on a different provider with a different EVPN network between the VMs

I need link all VMs between two cluster with two EVPN network

- All VMs from cluster 1 can communicate with VMs from cluster 2 and vice versa
- If cluster 1 or some nodes from it went down then network of cluster 2 should work (VMs from cluster 2 are still communicating with each other), it just can't communicate with VMs from cluster 1

What is best for this?

 

[spirit]

Do you really need 2 differents evpn network ?

if your proxmox hypervisors on the differents sites can communicate directly, you could use same evpn network, and do a full mesh (defined all proxmox nodes as peers on both side).


If you have really 2 differents evpn networks, with 2 asn, it's possible to link them , but it's need a little be more setup. (Depend if all nodes can communicate directly together, or if you want only some gateway nodes able to communicate each other...)

 

[nikordev]

In cluster, all nodes communicate through VLANs
Between clusters nodes can communicate via WAN

For full mesh i need for evpn controller on both clusters set in pears all node from both clusters?

Example VLAN+WAN #1 (JSON format just for beauty ):

{
  "cluster1": {
    "nodes": {
      "node1": {
        "wan": "1.1.1.2",
        "vlan": "10.0.0.2"
      },
      "node2": {
        "wan": "1.1.1.3",
        "vlan": "10.0.0.3"
      },
      "node3": {
        "wan": "1.1.1.4",
        "vlan": "10.0.0.4"
      }
    },
    "asn": 65000,
    "pears": [
      "10.0.0.2",
      "10.0.0.3",
      "10.0.0.4",
      "2.2.2.2",
      "2.2.2.3",
      "2.2.2.4"
    ],
    "exit": [
      "node1"
    ]
  },
  "cluster2": {
    "nodes": {
      "node1": {
        "wan": "2.2.2.2",
        "vlan": "10.1.1.2"
      },
      "node2": {
        "wan": "2.2.2.3",
        "vlan": "10.1.1.3"
      },
      "node3": {
        "wan": "2.2.2.4",
        "vlan": "10.1.1.4"
      }
    },
    "asn": 65000,
    "pears": [
      "10.1.1.2",
      "10.1.1.3",
      "10.1.1.4",
      "1.1.1.2",
      "1.1.1.3",
      "1.1.1.4"
    ],
    "exit": [
      "node1"
    ]
  }
}

Or for a full mesh, I only need to use the WAN node address for the peers?

Example WAN #2

{
  "cluster1": {
    "nodes": {
      "node1": {
        "wan": "1.1.1.2",
        "vlan": "10.0.0.2"
      },
      "node2": {
        "wan": "1.1.1.3",
        "vlan": "10.0.0.3"
      },
      "node3": {
        "wan": "1.1.1.4",
        "vlan": "10.0.0.4"
      }
    },
    "asn": 65000,
    "pears": [
      "1.1.1.2",
      "1.1.1.3",
      "1.1.1.4",
      "2.2.2.2",
      "2.2.2.3",
      "2.2.2.4"
    ],
    "exit": [
      "node1"
    ]
  },
  "cluster2": {
    "nodes": {
      "node1": {
        "wan": "2.2.2.2",
        "vlan": "10.0.1.2"
      },
      "node2": {
        "wan": "2.2.2.3",
        "vlan": "10.0.1.3"
      },
      "node3": {
        "wan": "2.2.2.4",
        "vlan": "10.0.1.4"
      }
    },
    "asn": 65000,
    "pears": [
      "2.2.2.2",
      "2.2.2.3",
      "2.2.2.4",
      "1.1.1.2",
      "1.1.1.3",
      "1.1.1.4"
    ],
    "exit": [
      "node1"
    ]
  }
}

Vnets on two clusters must be copies of each other?

 

[spirit]

mmm, I don't known if you can mix private && public ips on both side, I never had tried it.
With only wan ips, I'm sure it'll works 100%.

I think you just need to tried your 2 examples to see if it's working. (your config is ok in your 2 examples).


for vnets && zones, you just need to have the same config. (you can copy /etc/pve/sdn/zones.cfg && /etc/pve/sdn/vnets.cfg)

 

[nikordev]

Thanks for the explanation, I'll try

 

[spirit]

I have looked a iittle bit, I don't mix that mixing private && public ip will work in a full mesh,
because the node need to announce evpn route, like "myvm ip address is behind the vtep(node) with this ip".
and you can't have 2 routes for same ip.

The only way if you want to use private ip on site1 and private ip on site2, is to have 2 differents evpn network, with different asn.
I have added some extras options in last sdn version, not yet avaible in gui.

in /etc/pve/zones.cfg, in your zone, you need to add a new option "'rt-import <remoteas:remotezonevxlan>"
and in sdn controllers, you need to add bgp controller for each node with remote peers

for site1 (as 65000)

/etc/pve/sdn/zones.cfg

evpn: evpnzone
    controller evpnctl
    vrf-vxlan 10000
    rt-import 65001:10001

/etc/pve/sdn/controllers.cfg

evpn: evpnctl
        asn 65000
        peers 10.0.0.2,10.0.0.3,10.0.0.4

bgp: bgpnode1
        asn 65000
        node formationkvm1
        peers 2.2.2.2,2.2.2.3,2.2.2.4

bgp: bgpnode2
        asn 65000
        node formationkvm1
        peers 2.2.2.2,2.2.2.3,2.2.2.4

bgp: bgpnode3
        asn 65000
        node formationkvm1
        peers 2.2.2.2,2.2.2.3,2.2.2.4

for site2 (as65001)

/etc/pve/sdn/zones.cfg

evpn: evpnzone
    controller evpnctl
    vrf-vxlan 10001
    rt-import 65000:10000

/etc/pve/sdn/controllers.cfg

evpn: evpnctl
        asn 65000
        peers 10.0.1.2,10.0.1.3,10.0.1.4

bgp: bgpnode1
        asn 65000
        node formationkvm1
        peers 1.1.1.2,1.1.1.3,1.1.1.4

bgp: bgpnode2
        asn 65000
        node formationkvm1
        peers 1.1.1.2,1.1.1.3,1.1.1.4

bgp: bgpnode3
        asn 65000
        node formationkvm1
        peers 1.1.1.2,1.1.1.3,1.1.1.4

(I don't have tested it, but I think it should works)


Of course if you can use wan only ip, it's simplest.

 

[nikordev]

Hello @spirit
Sorry for the long answer

I checked your configuration and my two VMs from different clusters can communicate but only from exitnodes in both clusters.
All other virtual machines cannot communicate with each other from one or the other cluster.
Successful communication only between 10.10.1.2 and 10.10.2.2.

Without BGP controllers, VMs in the same cluster successfully communicate with each other.

My test configuration:

Cluster 1:

Node c001-n008: 10.8.4.43 CT100: 10.10.1.2

Node c001-n009: 10.8.4.44 CT101: 10.10.1.3

/etc/pve/sdn/controllers.cfg

evpn: evpnctl
    asn 65000
    peers 10.8.4.43,10.8.4.44

bgp: bgpc001-n008
    asn 65001
    node c001-n008
    peers 10.8.4.45,10.8.4.46
    bgp-multipath-as-path-relax 0
    ebgp 1

bgp: bgpc001-n009
    asn 65001
    node c001-n009
    peers 10.8.4.45,10.8.4.46
    bgp-multipath-as-path-relax 0
    ebgp 1

/etc/pve/sdn/zones.cfg

evpn: evpnzone
    controller evpnctl
    vrf-vxlan 10000
    exitnodes c001-n008
    exitnodes-primary c001-n008
    ipam pve
    mac 92:2E:0C:17:13:C0
    mtu 1350
    rt-import 65001:10001

/etc/pve/sdn/vnets.cfg

vnet: vnet1
    zone evpnzone
    tag 11000

/etc/pve/sdn/subnets.cfg

subnet: evpnzone-10.1.1.0-24
    vnet vnet1
    gateway 10.1.1.1
    snat 1

Cluster 2:

Node c001-n010: 10.8.4.45 CT100: 10.10.2.2

Node c001-n011: 10.8.4.46 CT101: 10.10.2.3

/etc/pve/sdn/controllers.cfg

evpn: evpnctl
    asn 65001
    peers 10.8.4.45,10.8.4.46

bgp: bgpc001-n010
    asn 65000
    node c001-n010
    peers 10.8.4.43,10.8.4.44
    bgp-multipath-as-path-relax 0
    ebgp 1

bgp: bgpc001-n011
    asn 65000
    node c001-n011
    peers 10.8.4.43,10.8.4.44
    bgp-multipath-as-path-relax 0
    ebgp 1

/etc/pve/sdn/zones.cfg

evpn: evpnzone
    controller evpnctl
    vrf-vxlan 10001
    exitnodes c001-n010
    exitnodes-primary c001-n010
    ipam pve
    mac 4A:BA:49:B0:2F:03
    mtu 1350
    rt-import 65000:10000

/etc/pve/sdn/vnets.cfg

vnet: vnet1
    zone evpnzone
    tag 11000

/etc/pve/sdn/subnets.cfg

subnet: evpnzone-10.1.2.0-24
    vnet vnet1
    gateway 10.1.2.1
    snat 1

P.S.

I tried to use all nodes in exitnodes but it didn't help.

 

[spirit]

I'm currently on holiday. I ll come back in 2 weeks.