L2 loop?

C

cyruspy

Renowned Member
Jul 2, 2013
146
9
83
Hello,

I have a cluster with SDN configured, EVPN for East/West communication and regular BGP towards external network elements. This was working fine for some time, but today I can't receive traffic from external network in a VM (ping to a public IP doesn't get an answer, even though the external network element is sending the traffic to the host).

Reviewing the environment, I see the VM MAC registered in all the firewall bridges:

Bash: 
root@pve-01:~# bridge fdb show | grep bc:24:11:f4:89:b9
bc:24:11:f4:89:b9 dev tap100i0 master fwbr100i0
bc:24:11:f4:89:b9 dev fwpr100p0 master ol107003
bc:24:11:f4:89:b9 dev fwln104i0 master fwbr104i0
bc:24:11:f4:89:b9 dev fwln260i0 master fwbr260i0
bc:24:11:f4:89:b9 dev fwln261i0 master fwbr261i0
bc:24:11:f4:89:b9 dev fwln262i0 master fwbr262i0
bc:24:11:f4:89:b9 dev fwln263i0 master fwbr263i0
bc:24:11:f4:89:b9 dev fwln103i0 master fwbr103i0

Is this expected?, to me it seems like a L2 loop. Any hints on how to diagnose this?
 
after further review, I see forwarding is not enabled in any of my hosts:

Bash: 
root@pve-04:~# sysctl net.ipv4.conf.all.forwarding
net.ipv4.conf.all.forwarding = 0

I can't seen any mention on the documentation to explecitely enable forwarding for IPv4, not sure if this is handled underneath automatically (and stopped working?):

https://pve.proxmox.com/wiki/Software-Defined_Network#pvesdn_main_configuration
https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#_sdn_core

Enabling IPv4 forwarding, I can see answers but with lots of duplicated packets:


Code: 
64 bytes from 8.8.4.4: icmp_seq=679 ttl=113 time=53.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=65241 ttl=115 time=52.9 ms
64 bytes from 8.8.8.8: icmp_seq=65241 ttl=115 time=52.9 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=65241 ttl=115 time=52.9 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=680 ttl=113 time=53.2 ms
64 bytes from 8.8.4.4: icmp_seq=680 ttl=113 time=53.2 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=680 ttl=113 time=53.2 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=65242 ttl=115 time=52.9 ms
64 bytes from 8.8.8.8: icmp_seq=65242 ttl=115 time=52.9 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=65242 ttl=115 time=52.9 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=681 ttl=114 time=53.3 ms
64 bytes from 8.8.4.4: icmp_seq=681 ttl=114 time=53.3 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=681 ttl=114 time=53.3 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=682 ttl=114 time=53.1 ms
64 bytes from 8.8.4.4: icmp_seq=682 ttl=114 time=53.1 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=682 ttl=114 time=53.1 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=683 ttl=114 time=53.1 ms
64 bytes from 8.8.4.4: icmp_seq=683 ttl=114 time=53.1 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=683 ttl=114 time=53.1 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=684 ttl=114 time=53.2 ms
64 bytes from 8.8.4.4: icmp_seq=684 ttl=114 time=53.2 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=684 ttl=114 time=53.2 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=685 ttl=114 time=53.1 ms
64 bytes from 8.8.4.4: icmp_seq=685 ttl=114 time=53.1 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=685 ttl=114 time=53.1 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=686 ttl=114 time=53.1 ms
64 bytes from 8.8.4.4: icmp_seq=686 ttl=114 time=53.1 ms (DUP!)
64 bytes from 8.8.4.4: icmp_seq=686 ttl=114 time=53.1 ms (DUP!)

Any guidance would be greatly appreciated
 
Last edited:
Can you provide some additional information?

The configuration of the VM in question:

Code: 
qm config <VMID>

SDN config:

Code: 
cat /etc/pve/sdn/zones.cfg
cat /etc/pve/sdn/vnets.cfg
cat /etc/pve/sdn/subnets.cfg
cat /etc/pve/sdn/controllers.cfg

Network config of the host:

Code: 
cat /etc/network/interfaces
ip a
ip r
cat /etc/frr/frr.conf

Can you also provide a tcpdump of the ping traffic?

Code: 
tcpdump -envi any icmp

Information about the routing table of the EVPN zone:

Code: 
ip r show vrf vrf_<ZONENAME>
 
shanreich said:
Can you provide some additional information?

The configuration of the VM in question:

Code: 
qm config <VMID>

SDN config:

Code: 
cat /etc/pve/sdn/zones.cfg
cat /etc/pve/sdn/vnets.cfg
cat /etc/pve/sdn/subnets.cfg
cat /etc/pve/sdn/controllers.cfg

Network config of the host:

Code: 
cat /etc/network/interfaces
ip a
ip r
cat /etc/frr/frr.conf

Can you also provide a tcpdump of the ping traffic?

Code: 
tcpdump -envi any icmp

Information about the routing table of the EVPN zone:

Code: 
ip r show vrf vrf_<ZONENAME>
Click to expand...
should I disable the IPv4 forwarding again before collecting the sample?
 
cyruspy said:
should I disable the IPv4 forwarding again before collecting the sample?
Click to expand...
enabled, it needs to be enabled on exit nodes on the uplink for north / south traffic to work properly.

Can you also additionally post your current frr version? Did you recently update the cluster / make any changes to the network topology?
 
Last edited:
shanreich said:
enabled, it needs to be enabled on exit nodes on the uplink for north / south traffic to work properly.

Can you also additionally post your current frr version? Did you recently update the cluster / make any changes to the network topology?
Click to expand...
Will share once I reach home. The forwarding was configured in my troubleshooting. it was disabled. I believe exit nodes are not declared, the north/south setup was manual via frr+BGP.
 
You must log in or register to reply here.
Top Bottom