Hi all,
currently I am trying to set up a cluster with SDN with an EVPN controller using multiple exit nodes, but I can't get around an issue with a routing loop.
The packets I send are stuck in a loop between both exit nodes and never forwarded outside of the cluster.
Each exit node has a default route to the other exit node, forwarding the packet to the other node instead of sending it outside the cluster.
I confirmed this using tcpdump on both nodes, showing the same packet (sequence number) over and over again (sometimes also TTL exceeded messages):
So my question is: Is this mistake in my configuration oder a bug in the current PVE Version (8.1.3)? What can I do to fix this?
I already found this old commit (April 2022), that seems to fix this issue. The configuration that has been added in this commit seems to be present in my vtysh running config (see attached file), but I'm not familiar with this configuration.
For rerefence i attached the complete sdn configuration from /etc/pve/sdn/.
Overview of nodes and IPs
Routes on all nodes
RP-Filter
Just to be sure, i checked the RP-Filter config, it is disabled on all nodes:
currently I am trying to set up a cluster with SDN with an EVPN controller using multiple exit nodes, but I can't get around an issue with a routing loop.
The packets I send are stuck in a loop between both exit nodes and never forwarded outside of the cluster.
Each exit node has a default route to the other exit node, forwarding the packet to the other node instead of sending it outside the cluster.
I confirmed this using tcpdump on both nodes, showing the same packet (sequence number) over and over again (sometimes also TTL exceeded messages):
Code:
root@pve-red-01:~# tcpdump -i any icmp
[...]
11:24:21.140478 vrfvx_redzone In IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140478 vrfbr_redzone In IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140488 vrfbr_redzone Out IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140490 vrfvx_redzone Out IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140535 vrfvx_redzone In IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140535 vrfbr_redzone In IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140545 vrfbr_redzone Out IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140546 vrfvx_redzone Out IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140615 vrfvx_redzone In IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140615 vrfbr_redzone In IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140625 vrfbr_redzone Out IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140627 vrfvx_redzone Out IP 192.168.0.10 > 172.16.12.1: ICMP echo request, id 1442, seq 4, length 64
11:24:21.140768 vrfvx_redzone In IP 192.168.0.1 > 192.168.0.10: ICMP time exceeded in-transit, length 92
11:24:21.140768 vrfbr_redzone In IP 192.168.0.1 > 192.168.0.10: ICMP time exceeded in-transit, length 92So my question is: Is this mistake in my configuration oder a bug in the current PVE Version (8.1.3)? What can I do to fix this?
I already found this old commit (April 2022), that seems to fix this issue. The configuration that has been added in this commit seems to be present in my vtysh running config (see attached file), but I'm not familiar with this configuration.
For rerefence i attached the complete sdn configuration from /etc/pve/sdn/.
Overview of nodes and IPs
| Host | IP |
|---|---|
| Firewall | 172.16.12.1/25 |
| pve-red-01 | 172.16.12.11/25 |
| pve-red-02 | 172.16.12.12/25 |
| pve-red-03 | 172.16.12.13/25 |
| Test VNet Gateway (SDN VNet Config) | 192.168.0.1/24 |
| Test VM | 192.168.0.10/24 |
Routes on all nodes
Code:
root@pve-red-01:/tmp/sdn# vtysh -c "sh ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
B>* 0.0.0.0/0 [200/0] via 172.16.12.12, vrfbr_redzone (vrf vrf_redzone) onlink, weight 1, 00:24:50
C>* 172.16.12.0/25 is directly connected, vmbr0, 00:25:26
C>* 172.16.12.128/25 is directly connected, bond1, 00:25:24
B>* 192.168.0.0/24 [20/0] is directly connected, test (vrf vrf_redzone), weight 1, 00:25:27
root@pve-red-02:~# vtysh -c "sh ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
B>* 0.0.0.0/0 [200/0] via 172.16.12.11, vrfbr_redzone (vrf vrf_redzone) onlink, weight 1, 00:06:00
C>* 172.16.12.0/25 is directly connected, vmbr0, 00:06:36
C>* 172.16.12.128/25 is directly connected, bond1, 00:06:39
B>* 192.168.0.0/24 [20/0] is directly connected, test (vrf vrf_redzone), weight 1, 00:06:39
root@pve-red-03:~# vtysh -c "sh ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
C>* 172.16.12.0/25 is directly connected, vmbr0, 00:06:44
C>* 172.16.12.128/25 is directly connected, bond1, 00:06:46RP-Filter
Just to be sure, i checked the RP-Filter config, it is disabled on all nodes:
Code:
root@pve-red-01:~# sysctl -a | grep -P "net.ipv4.conf.(default|all).rp_filter"
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0