SDN - Bug multiple VRFs

rpcomaite · Apr 7, 2023

Hello everybody,

I noticed a bug when using multiple VRFs.
When using the same exit-nodes on different vrfs but the primary is different, there is a deny route-map that is added in the MAP_VTEP_IN route-map for type 5 routes.
So if vrf "red" has primary exit-node node 1 and vrf "blue" has primary exit-node node 2, node 1 will not receive the default route for vrf "blue".
Is this a bug or a voluntary configuration choice?

Thanks for your help

spirit · Apr 7, 2023

rpcomaite said:
Hello everybody,

I noticed a bug when using multiple VRFs.
When using the same exit-nodes on different vrfs but the primary is different, there is a deny route-map that is added in the MAP_VTEP_IN route-map for type 5 routes.
So if vrf "red" has primary exit-node node 1 and vrf "blue" has primary exit-node node 2, node 1 will not receive the default route for vrf "blue".
Is this a bug or a voluntary configuration choice?

Thanks for your help

mmmm, seem to be a bug. you can share the generated /etc/frr/frr.conf ?

rpcomaite · Apr 7, 2023

Here is the content of the conf file.

Code:

!
frr version 8.2.2
frr defaults datacenter
hostname core501
log syslog informational
service integrated-vtysh-config
!
vrf vrf_client1
 vni 2
exit-vrf
!
vrf vrf_client2
 vni 4
exit-vrf
!
router bgp 65000
 bgp router-id 10.250.10.1
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 10.250.10.2 peer-group VTEP
 neighbor 10.250.10.3 peer-group VTEP
 !
 address-family ipv4 unicast
  import vrf vrf_client1
  import vrf vrf_client2
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_client1
  import vrf vrf_client2
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_client1
 bgp router-id 10.250.10.1
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
router bgp 65000 vrf vrf_client2
 bgp router-id 10.250.10.1
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
 match evpn route-type prefix
 match evpn vni 4
 set metric 200
exit
!
route-map MAP_VTEP_OUT permit 2
exit
!
end

rpcomaite · Apr 7, 2023

node2:

Code:

frr version 8.2.2
frr defaults datacenter
hostname core502
log syslog informational
service integrated-vtysh-config
!
vrf vrf_client1
 vni 2
exit-vrf
!
vrf vrf_client2
 vni 4
exit-vrf
!
router bgp 65000
 bgp router-id 10.250.10.2
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 10.250.10.1 peer-group VTEP
 neighbor 10.250.10.3 peer-group VTEP
 !
 address-family ipv4 unicast
  import vrf vrf_client1
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_client1
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_client1
 bgp router-id 10.250.10.2
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
router bgp 65000 vrf vrf_client2
 bgp router-id 10.250.10.2
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast
  advertise ipv6 unicast
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
 match evpn route-type prefix
 match evpn vni 2
 set metric 200
exit
!
route-map MAP_VTEP_OUT permit 2
exit
!
end

spirit · Apr 7, 2023

mmm, ok

the current code is (done for each zone/vrf)

Code:

    if ($is_gateway) {

        if (!$exitnodes_primary || $exitnodes_primary eq $local_node) {
            #filter default type5 route coming from other exit nodes on primary node or both nodes if no primary is defined.
            my $routemap_config = ();
            push @{$routemap_config}, "match evpn route-type prefix";
            my $routemap = { rule => $routemap_config, action => "deny" };
            unshift(@{$config->{frr_routemap}->{'MAP_VTEP_IN'}}, $routemap);

This is to avoid, than an exit-node receive default routes from other exit-node, and forward the traffic in loop.

I need to do tests on my side, I'm not sure how it's works with differents vrf,
maybe adding a simple "match evpn vni X" could works ?

can you try to modifiy /etc/frr/frr/conf ?

Code:

route-map MAP_VTEP_IN deny 1
 match evpn vni 2
 match evpn route-type prefix
exit

and do a "systemctl reload frr"

?

(and if it's working, it could be great to test with 3 zones)

rpcomaite · Apr 7, 2023

it doesn't work.
What information would you need?

spirit · Apr 7, 2023

I'll try to reproduce in my lab next week.

can you create a bugreport on https://bugzilla.proxmox.com ?

rpcomaite · Apr 7, 2023

Ok, i'll do that.

rpcomaite · Apr 7, 2023

@spirit Why did you choose frr rather than OVN?

rpcomaite · Apr 7, 2023

Ticket created!
https://bugzilla.proxmox.com/show_bug.cgi?id=4657

spirit · Apr 7, 2023

rpcomaite said:
@spirit Why did you choose frr rather than OVN?

Because I'm using evpn in my datacenter on physical hardware (arista && mellanox), and evpn implementation in frr and vxlan in kernel in mainly maintenaid by cumulus/mellanox/nvidia (so production ready). and I really like standard, without need to depend on a specific vendor implementation.

Note than technically, It's possible to add new kind of zones and controllers (ovn could be possible). But I really don't have time for this.
The plan is to add the possibility of external sdn module, like for storage, when sdn will be out of beta.

rpcomaite · Apr 7, 2023

spirit said:
Because I'm using evpn in my datacenter on physical hardware (arista && mellanox), and evpn implementation in frr and vxlan in kernel in mainly maintenaid by cumulus/mellanox/nvidia (so production ready). and I really like standard, without need to depend on a specific vendor implementation.

Note than technically, It's possible to add new kind of zones and controllers (ovn could be possible). But I really don't have time for this.
The plan is to add the possibility of external sdn module, like for storage, when sdn will be out of beta.

Ok I understand better the reason.
Anyway, thank you very much for the good work done on the SDN module!

rpcomaite · Apr 7, 2023

spirit said:
mmm, ok

the current code is (done for each zone/vrf)

Code:

if ($is_gateway) { if (!$exitnodes_primary || $exitnodes_primary eq $local_node) { #filter default type5 route coming from other exit nodes on primary node or both nodes if no primary is defined. my $routemap_config = (); push @{$routemap_config}, "match evpn route-type prefix"; my $routemap = { rule => $routemap_config, action => "deny" }; unshift(@{$config->{frr_routemap}->{'MAP_VTEP_IN'}}, $routemap);

This is to avoid, than an exit-node receive default routes from other exit-node, and forward the traffic in loop.

I need to do tests on my side, I'm not sure how it's works with differents vrf,
maybe adding a simple "match evpn vni X" could works ?

can you try to modifiy /etc/frr/frr/conf ?

Code:

route-map MAP_VTEP_IN deny 1 match evpn vni 2 match evpn route-type prefix exit

and do a "systemctl reload frr"

?

(and if it's working, it could be great to test with 3 zones)

@spirit OK, I checked again and it actually works!

NODE1:

Code:

frr version 8.2.2
frr defaults datacenter
hostname core501
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_client2
 vni 4
exit-vrf
!
vrf vrf_client1
 vni 2
exit-vrf
!
router bgp 65000
 bgp router-id 10.250.10.1
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 10.250.10.2 peer-group VTEP
 neighbor 10.250.10.3 peer-group VTEP
 neighbor 10.250.10.250 peer-group VTEP
 !
 address-family ipv4 unicast
  import vrf vrf_client1
  import vrf vrf_client2
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_client1
  import vrf vrf_client2
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_client1
 bgp router-id 10.250.10.1
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
router bgp 65000 vrf vrf_client2
 bgp router-id 10.250.10.1
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match evpn vni 2
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
 match evpn vni 4
 match evpn route-type prefix
 set metric 200
exit
!
route-map MAP_VTEP_OUT permit 2
exit
!
line vty
!

NODE 2

Code:

frr version 8.2.2
frr defaults datacenter
hostname core502
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_client2
 vni 4
exit-vrf
!
vrf vrf_client1
 vni 2
exit-vrf
!
router bgp 65000
 bgp router-id 10.250.10.2
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 10.250.10.1 peer-group VTEP
 neighbor 10.250.10.3 peer-group VTEP
 neighbor 10.250.10.250 peer-group VTEP
 !
 address-family ipv4 unicast
  import vrf vrf_client1
  import vrf vrf_client2
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_client1
  import vrf vrf_client2
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_client2
 bgp router-id 10.250.10.2
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
router bgp 65000 vrf vrf_client1
 bgp router-id 10.250.10.2
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
 match evpn vni 4
 match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
 match evpn vni 2
 match evpn route-type prefix
 set metric 200
exit
!
route-map MAP_VTEP_OUT permit 2
exit
!
line vty
!

rpcomaite · Apr 11, 2023

@spirit Here is the change I made to the code in the EvpnPlugin.pm file:

Code:

if (!$exitnodes_primary || $exitnodes_primary eq $local_node) {
     #filter default type5 route coming from other exit nodes on primary node or both >
      my $routemap_config = ();
      push @{$routemap_config}, "match evpn vni $vrfvxlan";
      push @{$routemap_config}, "match evpn route-type prefix";
      my $routemap = { rule => $routemap_config, action => "deny" };
      unshift(@{$config->{frr_routemap}->{'MAP_VTEP_IN'}}, $routemap);
} elsif ($exitnodes_primary ne $local_node) {
      my $routemap_config = ();
      push @{$routemap_config}, "match evpn vni $vrfvxlan";
      push @{$routemap_config}, "match evpn route-type prefix";
      push @{$routemap_config}, "set metric 200";
      my $routemap = { rule => $routemap_config, action => "permit" };
      unshift(@{$config->{frr_routemap}->{'MAP_VTEP_OUT'}}, $routemap);
}

spirit · Apr 11, 2023

Hi,

yes , I have done tests today on my lab, was able to reproduce.
and

push @{$routemap_config}, "match evpn vni $vrfvxlan";

in deny route-map , indeed fix it for me too.

I'll send a patch to the pve-devel mailing list.

Thanks for the report !

Search

Search

SDN - Bug multiple VRFs

rpcomaite

New Member

spirit

Distinguished Member

rpcomaite

New Member

rpcomaite

New Member

spirit

Distinguished Member

rpcomaite

New Member

spirit

Distinguished Member

rpcomaite

New Member

rpcomaite

New Member

rpcomaite

New Member

spirit

Distinguished Member

rpcomaite

New Member

rpcomaite

New Member

rpcomaite

New Member

spirit

Distinguished Member