SDN EVPN: stale local path per VM address after a VXLAN interface is recreated (FRR 10.6.1-1+pve3)

Pascal Ehlert

New Member
Feb 12, 2025
3
2
3
Arnsberg, Germany
hostzero.com
Hi,

we ran into stale BGP paths in the EVPN zone VRF and can reproduce them. I have not found an existing report, so here it is. Names and addresses are replaced: node8 is the test host (VTEP 192.0.2.108), vnet1 the VNet (VNI 10244, 10.0.44.0/24), 10.0.44.100 a VM on node8.

Versions
pve-manager 9.2.21, kernel 7.0.14-25-pve
frr and frr-pythontools 10.6.1-1+pve3
libpve-network-perl 1.6.7, ifupdown2 3.3.0-1+pmx12

Setup
9 nodes, one EVPN zone (vrf-vxlan 1000, all nodes are exit nodes, disable-arp-nd-suppression 1), 18 VNets with subnets and gateways. EVPN controller with an iBGP full mesh (AS 65100), plus a BGP controller on three nodes that peer with an external router over eBGP. The FRR config is the generated one, unmodified.

What happens
If the VXLAN interface of a VNet is deleted and created again while FRR keeps running, bgpd ends up with two paths in the zone VRF for every local host route (type-2 MAC/IP) of that VNet. One is the import from before the interface was recreated, now flagged "inaccessible, import-check enabled". The other is a fresh import of the same EVPN route.

The old path is never removed. It is still there after the EVPN route has been withdrawn, for example when the VM or a virtual IP moves to another node. In the default table (import vrf vrf_default) the old path wins best-path selection, so the node keeps a locally sourced best path for an address it no longer hosts. On nodes with a BGP controller that is the path announced to the external peer. Only a restart of frr clears it.

We got this on every node at once from an SDN apply after changing a zone option. ifreload removed and re-added all vxlan_<vnet> interfaces (kernel log "vxlan_<vnet> (unregistering)", zebra "netlink-dp ... error: No such device, type=RTM_NEWNEIGH"). Afterwards every local address on every node had the extra path.

Steps to reproduce
On a node with at least one running VM in an EVPN VNet:

1. systemctl restart frr, wait for the EVPN sessions.
2. This prints 0, and each local address has one path:
Code:
vtysh -c 'show bgp vrf vrf_default ipv4 unicast detail' | grep -c 'inaccessible, import-check'
3. Recreate the interface. It is back and attached to the VNet bridge within a second:
Code:
ip link del vxlan_vnet1; ifreload -a
4. Run the command from step 2 again. It now counts one line per local address in vnet1. Addresses in other VNets are unchanged.

Output after step 3
10:04:00 is the FRR start, 10:04:49 the moment the interface was recreated.

Code:
# vtysh -c 'show bgp vrf vrf_default ipv4 unicast 10.0.44.100/32'
BGP routing table entry for 10.0.44.100/32, version 0
Paths: (2 available, no best path)
  Not advertised to any peer
  Imported from 192.0.2.108:16:[2]:[0]:[48]:[aa:bb:cc:00:00:01]:[32]:[10.0.44.100], VNI 10244/1000  Local
    192.0.2.108(node8) (inaccessible, import-check enabled) from node8(Static announcement) (0.0.0.0) announce-nh-self
      Origin IGP, weight 32768, valid, external
      Last update: Fri Nov 20 10:04:00 2026
  Imported from 192.0.2.108:16:[2]:[0]:[48]:[aa:bb:cc:00:00:01]:[32]:[10.0.44.100], VNI 10244/1000  Local
    192.0.2.108(node8) from node8(Static announcement) (0.0.0.0) announce-nh-self
      Origin IGP, weight 32768, valid, external
      Last update: Fri Nov 20 10:04:49 2026

# vtysh -c 'show bgp ipv4 unicast 10.0.44.100/32'
BGP routing table entry for 10.0.44.100/32, version 86
Paths: (2 available, best #1, table default)
  Imported from 192.0.2.108:2:10.0.44.100/32
  Local
    192.0.2.108(node8) (inaccessible, import-check enabled) from 0.0.0.0 (192.0.2.108) vrf vrf_default(12) announce-nh-self
      Origin IGP, weight 32768, valid, sourced, local, multipath, best (Locally configured route)
      Last update: Fri Nov 20 10:04:00 2026
  Imported from 192.0.2.108:2:10.0.44.100/32
  Local
    192.0.2.108(node8) from 0.0.0.0 (192.0.2.108) vrf vrf_default(12) announce-nh-self
      Origin IGP, weight 32768, valid, sourced, local, multipath, bestpath-from-AS Local
      Last update: Fri Nov 20 10:04:49 2026

The EVPN table itself has the route once (show bgp l2vpn evpn route rd 192.0.2.108:16 type macip shows 1 path, best).

After the address has moved to another node
We saw this on four nodes for four different addresses, days after the SDN apply. Here 10.0.44.200 moved from node6 to node2:

Code:
# vtysh -c 'show bgp vrf vrf_default ipv4 unicast 10.0.44.200/32'
Paths: (2 available, best #1, vrf vrf_default)
  Imported from 192.0.2.102:16:[2]:[0]:[48]:[aa:bb:cc:00:00:02]:[32]:[10.0.44.200], VNI 10244/1000  Local
    192.0.2.102(node2) from node2(192.0.2.102) (192.0.2.102) announce-nh-self
      Origin IGP, localpref 100, valid, internal, best (First path received)
  Imported from 192.0.2.106:16:[2]:[0]:[48]:[aa:bb:cc:00:00:03]:[32]:[10.0.44.200], VNI 10244/1000  Local
    192.0.2.106(node6) (inaccessible, import-check enabled) from node6(Static announcement) (0.0.0.0) announce-nh-self
      Origin IGP, weight 32768, valid, external
      Last update: (from before the SDN apply)

# vtysh -c 'show bgp ipv4 unicast 10.0.44.200/32'
Paths: (2 available, best #1, table default)
    192.0.2.106(node6) (inaccessible, import-check enabled) from 0.0.0.0 (192.0.2.106) vrf vrf_default(12) announce-nh-self
      Origin IGP, weight 32768, valid, sourced, local, bestpath-from-AS Local, best (Weight)
    192.0.2.102(node6) from 0.0.0.0 (192.0.2.106) vrf vrf_default(12) announce-nh-self
      Origin IGP, localpref 100, valid, sourced, local

The EVPN route that the second path claims to be imported from (RD 192.0.2.106:16) is gone from show bgp l2vpn evpn route. The main routing table on node6 has no /32 for the address, while the other nodes have "10.0.44.200 via 192.0.2.102 dev vrfbr_default". The lookup on node6 falls back to the leaked /24.

Expected
One path per local MAC/IP route in the VRF after the interface is recreated, and none left once the EVPN route is withdrawn.

Impact
The extra paths pile up unnoticed until an address moves. Nothing in the GUI or in the BGP summary points at them.

We had a partial outage for a virtual IP after it moved away from a node that peers with the external router. That node still had the stale local path as best and announced it, and restarting frr there fixed it immediately. I could not capture where the packets were dropped. In later tests, nodes in the same BGP state forwarded host-originated traffic to the moved address correctly through the VRF. So I can only say the outage coincided with the stale path, not that the path alone caused it.

Workaround
systemctl restart frr on each node after an SDN apply that recreated VXLAN interfaces. The EVPN sessions were back within 6 to 30 seconds.

Is this known, or should I open a Bugzilla entry for it?
 
Follow-up: I found what actually drops the traffic, and it is a second problem next to the extra BGP path from my first post. Same anonymisation as above; node5 is 192.0.2.105, node6 is 192.0.2.106, and 10.0.44.200 is a Kubernetes load balancer address (Cilium L2 announcements): the Kubernetes node that holds a lease for it answers ARP for the address. The Kubernetes nodes are VMs on the PVE nodes.

What happened
I restarted frr on all nodes, one at a time, to get rid of the extra paths. No VM was migrated. The frr restart on node6 cut the Kubernetes node running there off from its API server for about five seconds. It could not renew the lease for the address (the Cilium agent log shows the failed renewal), and two seconds later the Kubernetes node on node5 took the lease over and started answering ARP. So the address changed PVE nodes while both VMs stayed where they were. From then on every external connection to the address hung: the TCP handshake completed, the TLS Client Hello never got an answer. Requests from a PVE node itself (source = underlay address) still worked.

node5 had this in its main routing table, although it was now the owner of the address:

Code:
# ip route show table main 10.0.44.200
10.0.44.200 nhid 1343 via 192.0.2.106 dev vrfbr_default proto bgp metric 20 onlink

# vtysh -c 'show ip route'   (default VRF, taken 5.5 minutes after the move)
B>* 10.0.44.200/32 [200/0] via 192.0.2.106, vrfbr_default (vrf vrf_default) onlink, weight 1, 00:05:30

bgpd on node5 no longer had that path. Both tables only held the new local one:

Code:
# vtysh -c 'show bgp ipv4 unicast 10.0.44.200/32'
Paths: (1 available, best #1, table default)
  Imported from 192.0.2.105:2:10.0.44.200/32
  Local
    192.0.2.105(node5) from 0.0.0.0 (192.0.2.105) vrf vrf_default(12) announce-nh-self
      Origin IGP, weight 32768, valid, sourced, local, bestpath-from-AS Local, best (First path received)
      Last update: Fri Nov 20 10:21:32 2026

# vtysh -c 'show bgp vrf vrf_default ipv4 unicast 10.0.44.200/32'
Paths: (1 available, no best path)
  Imported from 192.0.2.105:18:[2]:[0]:[48]:[aa:bb:cc:00:00:05]:[32]:[10.0.44.200], VNI 10244/1000  Local
    192.0.2.105(node5) from node5(Static announcement) (0.0.0.0) announce-nh-self
      Origin IGP, weight 32768, valid, external
      Last update: Fri Nov 20 10:21:32 2026

So zebra kept a route that bgpd had replaced three seconds after it was installed. The order of events on node5:

10:20:01 frr restarted on node5 (so this is a freshly started frr)
10:21:23 node6 restarts frr, session down
10:21:28 session to node6 up again, node6 announces 10.0.44.200, node5 installs the /32 via node6
10:21:30 the Kubernetes node on node6 fails to renew its lease for the address
10:21:32 the Kubernetes node on node5 takes the lease and answers ARP, bgpd on node5 switches to the local path
after that: the kernel route via node6 stays until frr is restarted on node5

Why this drops traffic
Our external router sends packets for the address straight to node5, because the exit nodes announce the /32 with the owner's underlay address as next hop. node5 looks it up in the main table, finds the stale /32 and sends the packet over the L3 VNI to node6. node6 knows the address is on node5 and sends it back. In tcpdump on node5 the SYN goes out on vrfvx_default and comes back in, and is delivered to the VM. Every later packet of the connection goes out on vrfvx_default and never comes back. My guess is that node6 drops them as conntrack INVALID in the PVE firewall forward chain, since it only ever sees one direction, but I did not capture on node6.

On a node that is the owner and has no such stale route, the main table has no /32 for the address at all and the lookup falls through to the leaked /24 into the VRF, which works.

Not the first time
We had exactly this state twice a few weeks ago after an address or a VM changed nodes, without any frr restart involved at that time: bgpd correct, kernel /32 in the main table still pointing at the previous node, fixed by restarting frr on the new owner. So the frr restart on node6 was only what made the lease, and with it the address, change nodes in this case.

I can't trigger this one on demand yet. What I use to find it after a move: for every local type-2 address, the main table must not have a /32 via another node.

Code:
for ip in $(vtysh -c 'show evpn arp-cache vni all' | awk '$2=="local"{print $1}' | grep -v :); do ip route show table main "$ip/32" | grep via; done

Workaround is again systemctl restart frr, this time on the node the address moved to.
 
Last edited: