[SOLVED] EVPN - traffic not passing

[cwathan]

Has anyone had an issue with EVPNs not passing traffic? I created a VNET that is meant to be PtP. I configured an IP address on the associated VM internet (within the same /31) and traffic fails in both directions. ARP isn't being learned. However a 'show interface' within vtysh on both proxmox hosts show the vxlan interfaces as up/up. The traffic has no need to exit the VXLAN. What am I missing?

Controller: EVPN
EVPN VRF-VXLAN Tag: 100
Exit Nodes: None
Primary Exit Node: N/A
Nodes: All

VNET Zone: Name of Zone with VRF-VXLAN Tag of 100
Tag: 105

Subnets: Currently not configured but, had been previously configured for the VNET zone with the /31 of the VM PtP interfaces

 

[spirit]

mmm, I never had tried with /31.
but with /31, you don't have broadcast address , so no arp, so no learning is possible from frr.

you should configure subnet (/30 minimum) with gateway, to have the anycast gateway defined on the vnet, then use this gateway in your vm. (so traffic will go
through the vm, mac will be listened in the vnet && ip in the host arp table, then frr will be able to learn.

Do you really need /31 ?

 

[cwathan]

spirit, this is for a PtP connection between two virtual routers. A /31 conserves IP space and has been the generally accepted norm for IPv4 Point to Point (PtP) links for many years. So, in a day and age when you have to purchase IPv4 space second hand at exhorbinate rates ($40-$65/IP) it is highly preferred to not waste IPs on a PtP.

Are you suggesting my work around would be to manually add the ARP entry within the VMs?

 

[spirit]

spirit, this is for a PtP connection between two virtual routes. A /31 conserves IP space and has been the generally accepted norm for IPv4 Point to Point (PtP) links for many years. So, in a day and age when you have to purchase IPv4 space second hand at exhorbinate rates ($40-$65/IP) it is highly preferred to not waste IPs on a PtP.

Are you suggesting my work around would be to manually add the ARP entry within the VMs?

Maybe it could work. (I'm planning to static add arp entry in future at vm start, but currently it's really done with listening).

but you really need an gateway ip on the vnet for the routing.

Just one question: "The traffic has no need to exit the VXLAN". Is it public ipv4 ? I don't understand why you don't need to exit the vxlan in this case ?

What is your usecase with evpn here ? (vs simple routing)

 

[cwathan]

spirit, no traffic needs to leave the VNET because it is a cross-connect between the two router VMs for optimal routing. The traffic will only ever go between the two VMs and nowhere else. The VMs have physical interfaces shared to them that connect to independent providers. The VMs also have another VNIC that is a simple vmbr VLAN to the common backbone.

A gateway should never be needed if your traffic isn't exiting the VXLAN. Would you want a customer's private VXLAN to have the potential to talk to another customer's private VXLAN because they both had gateways assigned and the routes were injected into the BGP table?

 

[spirit]

spirit, no traffic needs to leave the VNET because it is a cross-connect between the two router VMs for optimal routing. The traffic will only ever go between the two VMs and nowhere else. The VMs have physical interfaces shared to them that connect to independent providers. The VMs also have another VNIC that is a simple vmbr VLAN to the common backbone.

ah ok , got it.

so, the 2 routers vm are on the same vnet ? if yes, Can't you use a simple vxlan zone instead evpn ? evpn seem to be overkill in your usecase.


(evpn could be usefull to replace your vm routers,with proxmox host acting as exit-node for example).

but here, it's look like you need a layer2 network between your 2 routing vm.

A gateway should never be needed if your traffic isn't exiting the VXLAN. Would you want a customer's private VXLAN to have the potential to talk to another customer's private VXLAN because they both had gateways assigned and the routes were injected into the BGP table?

(with evpn, you can create multiples zone, they are differents vrf with different routing tables)

 

[cwathan]

My understanding of your SDN deployment is that the VNETs are the VXLANs within an EVPN. Is that incorrect?

I've statically assigned the ARP entries and traffic still is not passing through. I've created an independent VXLAN that is not under the EVPN and attempted the same. It still is not passing traffic. At one point, I had found output that the VNET interfaces were actually dropping all traffic. However, I don't recall where I saw that. I'm really not sure what would cause that behavior in a virtualized interface with no FW or filtering enabled.

 

[spirit]

My understanding of your SDN deployment is that the VNETs are the VXLANs within an EVPN. Is that incorrect?

a vnet (in evpn zone or vxlan zone type) is a bridge with a vxlan interface plugged inside the bridge.

With a simple vxlan zone, it's like a simple tunnel between remote bridge through the vxlan. (arp/bum traffic is flooded to vxlan port to learn mac address)

with evpn, you have frr router, is listening to mac && ips address, and inject mac/ip in differents nodes. (and do also routing)

So if you don't need to route with anycast gateway and all evpn features, use a simple vxlan zone

 

[spirit]

I just tested between 2 vms, with a vxlan zone, and same vnet, on 2 different proxmox nodes

vm1 : 192.168.0.10/31 (no gateway)
vm2 : 192.168.0.11/31 (no gateway)

I can ping between both vm without problem.

 

[cwathan]

I created a VXLAN zone and created a VNET within that zone. I assigned the VNICs to the VXLAN VNET (selecting just the VXLAN zone isn't permitted in the GUI). Traffic is still not passing.

I ended up having to use the standard vmbr bridge interface and add a VLAN to that bridge. There seems to be a problem with VXLAN config as well.

If it is helpful I can reconfigure back to the EVPN and VXLAN configs and paste their configs here.

 

[spirit]

I created a VXLAN zone and created a VNET within that zone. I assigned the VNICs to the VXLAN VNET (selecting just the VXLAN zone isn't permitted in the GUI). Traffic is still not passing.

I ended up having to use the standard vmbr bridge interface and add a VLAN to that bridge. There seems to be a problem with VXLAN config as well.

If it is helpful I can reconfigure back to the EVPN and VXLAN configs and paste their configs here.

Any proxmox firewall enabled on host, blocking vxlan ports ? (udp 4789)

It should really works out of the box with a simple vxlan zone.

 

[cwathan]

FW is disabled on grid, host and VM level and traffic is still not passing. I statically assigned ARP and traffic still does not pass.

 

[spirit]

can you send content of /etc/pve/sdn/*.cfg ,

+
/etc/network/interfaces , /etc/network/interfaces.d/sdn on each host

+
result of

#ip addr on each host ?

 

[cwathan]

spirit,

I apologize tit took me a while to get back to you on this. Attached is the information you requested.

 

[spirit]

Hi,

you should avoid to use ovs && bridge with vlan-aware in parallel when you use evpn. They are known breaking bug in frr (as frr is listening in kernel bridge netlink event). and ovs or vlan-aware bridge can break it.

 

[cwathan]

So, the solution would be to disable "vlan-aware" on all Vnets? I've tried that without success. Do I need to restart nodes or frr after this change? Or, did you mean I should not use OVS with SDN/frr? I'm beginning to think restarting from scratch and seeing what happens might be worthwhile.

 

[spirit]

Oh, I didn't see that you have also enable vlanaware on vnet.

1) disable vlan-aware on vnet. It's only used if you want to do vlan tag (in the vm nic gui) over vxlan. (some users requested it)

then try with a vxlan zone. (not evpn yet).

2) if you want to use an evpn zone, don't use ovs or vlan-aware bridge in your /etc/network/interfaces (for classic vmbrX)

 

[cwathan]

For VXLAN, do I need to enable VXLAN on the physical switch for traffic that is passing through between Proxmox nodes? I've been working under the presumption that it was not necessary for pass through traffic so long as MTU settings were correct. I know that you need to configure VXLAN on the physical switch if you want the physical switch to participate in any given VXLAN.

 

[spirit]

For VXLAN, do I need to enable VXLAN on the physical switch for traffic that is passing through between Proxmox nodes? I've been working under the presumption that it was not necessary for pass through traffic so long as MTU settings were correct. I know that you need to configure VXLAN on the physical switch if you want the physical switch to participate in any given VXLAN.

no. vxlan is an overlay network working over tcp-ip between your proxmox nodes.

 

[cwathan]

An update to everyone, after some additional digging I discovered that if you install the strongswan packet for VXLAN IPsec encryption, it automatically takes effect. I had installed the packet with the intent to configure and utilize after validating VXLAN worked without.

What this resulted in was stronswan attempting to turn up IPsec encryption between 2 nodes that had the VXLAN utilized. Since strongswan was not configured the IPsec tunnel would not establish and traffic could not pass.

There are two pieces to note here, and the Proxmox Wiki for SDN should be updated to call out... 1) Do not install the strongswan package PRIOR to confirming your VXLAN config is passing traffic. 2) You need to reload the /etc/ipsec.secrets file once you update it with a PSK. The command to do so is:

ipsec rereadsecrets

Once the config file is reloaded the daemon uses the configured PSK.

****Spirit, thank you for your time and patience helping me work through this!****