EVPN with multiple node interfaces

[wile.e.coyote]

Hi brains trust,

I have a 2 node cluster set up and working. Each node is connected to public internet via eno1 and to the other node (via a switch) via eno2. I have created two linux bridges vmbr0 and vmbr1, which bridge eno1 and eno2 respectively such that:
Node1:
- vmbr0: <public ip>
- vmbr1: 10.0.0.1/24
Node2:
- vmbr0: <public ip>
- vmbr1: 10.0.0.2/24
etc...

From the nodes I can ping the other node on either the public ip or the 10.0.0.x private address.

I currently have a working EVPN that is just using the public ip's as the peers. Any VM that attaches to a VNET in the EVPN is able to reach the public internet okay.

The problem with this set up is that all VM (node1)<-> VM(node2) traffic goes over the public internet which I want to avoid, instead I would like this routed via vmbr1. However, I still need each of the VM's to be able to route traffic to public internet.

I've tried (see config below) creating an EVPN controller with the vmbr1 ip's (10.0.0.1, 10.0.0.2) instead of the public ip's, and when creating the EVPN zone, I set node2 as the exit node. However whilst two VM's in this zone can communicate with each other, they can't ping public ip addresses - including the public ip address of the host node.

I'm sure I'm missing something obvious, and my networking knowledge is pretty basic, but I assume this would be a pretty common usecase? (keeping inter-vm traffic off public facing interface). Any assistance greatly appreciated! Let me know if you need any additional details.

NODE CONFIG:

Node /etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual


auto vmbr0
iface vmbr0 inet static
        address <public-ip>
        gateway <public-ip-gw>
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr1
iface vmbr1 inet static
        address 10.0.0.1/24
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

source /etc/network/interfaces.d/*

/etc/pve/sdn/controllers.cfg

evpn: myevpn
        asn 65000
        peers 10.0.0.1, 10.0.0.2

/etc/pve/sdn/zones.cfg

evpn: test
        controller myevpn
        vrf-vxlan 10000
        disable-arp-nd-suppression 1
        exitnodes node1,node2
        exitnodes-primary node2
        ipam pve
        mac <hidden>
        mtu 1350

/etc/pve/sdn/vnets.cfg

vnet: mynet
        zone test
        tag 10100

/etc/pve/sdn/subnets.cfg

subnet: test-10.20.20.0-24
        vnet mynet
        gateway 10.20.20.1
        snat 1

VM Config:

/etc/netplan/config.yaml

network:
    version: 2
    ethernets:
        eth0:
            addresses:
            - 10.20.20.101/32
            match:
                macaddress: <hidden>
            nameservers:
                addresses:
                - 8.8.8.8
            routes:
            -     on-link: true
                to: default
                via: 10.20.20.1
            set-name: eth0

 

[spirit]

Hi,
they are a bug currently with multiple evpn exit-nodes on pve8. (This is already fixed in the proxmox git, but pve-network package is not yet released).

Can you try with just 1 exit-node ?

 

[wile.e.coyote]

Thanks for response spirit.

I've changed the zone to just have node2 as the exit node, however ping is still hanging.
It might be worth noting, I had multiple exit-nodes defined and working (including pinging public ips) with the exiting evpn that I'm replacing.

I'm able to ping between two VM's on the different nodes via the EVPN subnet IPs, and I can ping the subnet gateway okay, but I'm unable to ping either of the nodes themselves, using the public ip address (vmbr0) or the 10.10.10.x address (vmbr1)
i.e.
VM1: evpn subnet ip: 10.20.20.101
VM2: evpn subnet ip: 10.20.20.102

From VM1:
ping 10.20.20.102 - works
ping 10.20.20.1 - works
ping 10.0.0.1 - doesn't work
ping 10.0.0.2 - doesn't work
ping <node1 public ip> - doesn't work
ping <node2 public ip> - doesn't work
ping <any public ip> - doesn't work

I'm wondering if it's the way I set up vmbr1? The nodes can communicate over vmbr1, but it doesn't have a gateway defined - mainly because it's just a switch on that interface so nothing is routed. Would this cause issues when trying to route from VM out to public internet?

Thanks

 

[spirit]

you can't ping the node ips from vm ip, because they are in 2 differents vrf (2 differents routing table, this is by design). The exception is when you enable on zone "enable local-node routing" or maybe through snat. (because is this case, the private source is translate to host public).


but, you should be able to ping an external ip if the snat is working correctly.

the snat occur only on the exit-node, so if you launch a ping from to vm to 8.8.8.8 for example, if you do a tcpdump on the exit-node on vmbr0,
you should see the ping + snat with the public of the the node as source (instead the private ip of the vm).

 

[wile.e.coyote]

Thanks Spirit,

Yes it makes sense now that it shouldn't be able to ping 10.0.0.1 or 10.0.0.2, but the public ones it should via the exit node.

Turns out it must have been the config on the VM's . I ended up rebuilding them out of desperation and now they can both ping public addresses and all is working well!

Sorry for dragging you down my rabbit hole.
Thanks again for your assistance.