PVE Host Access to EVPN Endpoints not working

A

AristonPost

Member
Feb 15, 2024
28
5
8
Zurich
Hey all. I have a 4-Node 9.2.25 Cluster running with EVPN VNets for the workload. Configured VNets have /28 Subnets configured with Gateway.
An external VyOS Router is configured as BGP-RR. This setup is working for everything that comes from the VyOS router or any other EVPN VNet.

But I can't access the EVPN VNet Endpoints from the PVE Host without specifying the Management IP as src. The kernel selects an inproper IP (Gateway IP of the first EVPN VNet Subnet) as src and as a result the communications fails.

Anyone else expecting this behaviour or have an Idea how to fix this?
 
Are you currently leaking the routes into the default routing table?
Can you post the current configuration state? You can retract IPs, but please do it in a fashion so I can correlate them between the configuration files.

Code: 
cat /etc/pve/sdn/.running-config

ip r
 
Hi Stefan,
Please find the configuration below.

root@pve-11:~# cat /etc/pve/sdn/.running-config
{"subnets":{"ids":{"home-172.16.11.16-28":{"vnet":"vnet1102","gateway":"172.16.11.17","type":"subnet"},"home-172.16.11.48-28":{"vnet":"vnet1104","type":"subnet","gateway":"172.16.11.49"},"home-172.16.11.64-28":{"vnet":"vnet1105","gateway":"172.16.11.65","type":"subnet"},"home-172.16.11.128-28":{"gateway":"172.16.11.129","type":"subnet","vnet":"vnet1109"},"home-172.16.11.96-28":{"type":"subnet","gateway":"172.16.11.97","vnet":"vnet1107"},"home-172.16.11.80-28":{"gateway":"172.16.11.81","type":"subnet","vnet":"vnet1106"},"home-172.17.2.0-24":{"type":"subnet","gateway":"172.17.2.1","vnet":"vnet1702"},"home-172.16.11.32-28":{"vnet":"vnet1103","gateway":"172.16.11.33","type":"subnet"},"home-172.16.11.0-28":{"vnet":"vnet1101","gateway":"172.16.11.1","type":"subnet"},"home-172.16.11.144-28":{"vnet":"vnet1110","type":"subnet","gateway":"172.16.11.145"},"home-172.16.11.112-28":{"vnet":"vnet1108","gateway":"172.16.11.113","type":"subnet"}}},"route-maps":{"ids":{}},"controllers":{"ids":{"bgppve-14":{"bgp-multipath-as-path-relax":0,"ebgp":1,"asn":65010,"peers":"172.16.54.1","node":"pve-14","type":"bgp"},"bgppve-13":{"bgp-multipath-as-path-relax":0,"ebgp":1,"node":"pve-13","type":"bgp","asn":65010,"peers":"172.16.54.1"},"evpn1":{"type":"evpn","asn":65010,"peers":"172.16.54.11,172.16.54.12,172.16.54.13,172.16.54.14"},"bgppve-11":{"peers":"172.16.54.1","asn":65010,"type":"bgp","node":"pve-11","ebgp":1,"bgp-multipath-as-path-relax":0},"bgppve-12":{"peers":"172.16.54.1","asn":65010,"node":"pve-12","type":"bgp","bgp-multipath-as-path-relax":0,"ebgp":1}}},"zones":{"ids":{"home":{"vrf-vxlan":11000,"exitnodes":{"pve-13":1,"pve-12":1,"pve-11":1,"pve-14":1},"type":"evpn","ipam":"pve","advertise-subnets":1,"mtu":1500,"controller":"evpn1","mac":"BC:24:11:63:3A:C2"}}},"fabrics":{"ids":{}},"version":26,"vnets":{"ids":{"vnet1109":{"zone":"home","tag":11109,"type":"vnet","alias":"Media Services"},"vnet1702":{"type":"vnet","alias":"Teddycloud","zone":"home","tag":11702},"vnet1101":{"tag":11101,"zone":"home","type":"vnet","alias":"Infrastructure Management"},"vnet1107":{"alias":"CI/CD","type":"vnet","tag":11107,"zone":"home"},"vnet1110":{"type":"vnet","alias":"File Services","zone":"home","tag":11110},"vnet1103":{"zone":"home","tag":11103,"alias":"Monitoriing","type":"vnet"},"vnet1108":{"alias":"Cloud Office","type":"vnet","zone":"home","tag":11108},"vnet1106":{"type":"vnet","alias":"Source Control","zone":"home","tag":11106},"vnet1105":{"type":"vnet","alias":"Load Balancer","zone":"home","tag":11105},"vnet1104":{"alias":"Automation","type":"vnet","zone":"home","tag":11104},"vnet1102":{"zone":"home","tag":11102,"alias":"Identity Management","type":"vnet"}}},"prefix-lists":{"ids":{}}}
Click to expand...

root@pve-11:~# ip r
default via 172.16.51.1 dev mgmt proto kernel onlink
213.xx.xx.xx/20 nhid 81 via 172.16.54.1 dev vtep proto bgp metric 20
172.16.0.0/24 nhid 81 via 172.16.54.1 dev vtep proto bgp metric 20
172.16.10.0/24 nhid 81 via 172.16.54.1 dev vtep proto bgp metric 20
172.16.11.0/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.2 nhid 78 via 172.16.54.12 dev vrfbr_home proto bgp metric 20 onlink
172.16.11.3 nhid 515 via 172.16.54.13 dev vrfbr_home proto bgp metric 20 onlink
172.16.11.16/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.18 nhid 515 via 172.16.54.13 dev vrfbr_home proto bgp metric 20 onlink
172.16.11.19 nhid 78 via 172.16.54.12 dev vrfbr_home proto bgp metric 20 onlink
172.16.11.20 nhid 78 via 172.16.54.12 dev vrfbr_home proto bgp metric 20 onlink
172.16.11.21 nhid 116 via 172.16.54.14 dev vrfbr_home proto bgp metric 20 onlink
172.16.11.32/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.48/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.64/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.80/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.96/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.112/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.128/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.11.130 nhid 116 via 172.16.54.14 dev vrfbr_home proto bgp metric 20 onlink
172.16.11.144/28 nhid 28 dev vrf_home proto bgp metric 20
172.16.51.0/24 dev mgmt proto kernel scope link src 172.16.51.11
172.16.52.0/24 dev drbd proto kernel scope link src 172.16.52.11
Click to expand...

As you can see, the routes are looking good. But when I try to reach 172.16.11.18 from pve-11 (172.16.5x.11), it doesn't work. When I do this from pve-13 which is the node running the vm and running the gateway it works flawlessly.

Default ping from a Node that's not running the target VM
root@pve-11:~# ping -c 3 172.16.11.18
PING 172.16.11.18 (172.16.11.18) 56(84) bytes of data.

--- 172.16.11.18 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2046ms
Click to expand...

Now forcing the management IP to be used, packet walks trough external router
root@pve-11:~# ping -I 172.16.51.11 -c 3 172.16.11.18
PING 172.16.11.18 (172.16.11.18) from 172.16.51.11 : 56(84) bytes of data.
64 bytes from 172.16.11.18: icmp_seq=1 ttl=63 time=0.841 ms
64 bytes from 172.16.11.18: icmp_seq=2 ttl=63 time=0.843 ms
64 bytes from 172.16.11.18: icmp_seq=3 ttl=63 time=0.616 ms

--- 172.16.11.18 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2050ms
rtt min/avg/max/mdev = 0.616/0.766/0.843/0.106 ms
Click to expand...

Default ping from the node that's running the target VM
root@pve-13:~# ping -c3 172.16.11.18
PING 172.16.11.18 (172.16.11.18) 56(84) bytes of data.
64 bytes from 172.16.11.18: icmp_seq=1 ttl=64 time=0.313 ms
64 bytes from 172.16.11.18: icmp_seq=2 ttl=64 time=0.280 ms
64 bytes from 172.16.11.18: icmp_seq=3 ttl=64 time=0.322 ms

--- 172.16.11.18 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.280/0.305/0.322/0.018 ms
Click to expand...

For reference this is my network configuration (pve-11, other nodes accordingly)
root@pve-11:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface nic0 inet manual
iface nic1 inet manual
iface nic2 inet manual
iface nic3 inet manual

auto bond0
iface bond0 inet manual
bond-slaves nic2 nic3
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
bond-lacp-rate 1
mtu 9000

auto bond1
iface bond1 inet manual
bond-slaves nic0 nic1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
bond-lacp-rate 1
mtu 1600

auto mgmt
iface mgmt inet static
address 172.16.51.11/24
gateway 172.16.51.1
vlan-id 51
vlan-raw-device bond0

auto drbd
iface drbd inet static
address 172.16.52.11/24
vlan-id 52
vlan-raw-device bond0

auto sync
iface sync inet static
address 172.16.53.11/24
vlan-id 53
vlan-raw-device bond0

auto vtep
iface vtep inet static
address 172.16.54.11/24
vlan-id 54
vlan-raw-device bond1

source /etc/network/interfaces.d/*
Click to expand...
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back