SDN FRR custom configuration

volga629 · Jun 7, 2022

Hello Everyone,
SDN feature works great with 7.2.3 version for EVPN East West traffic .
Is there are possibility preserve custom FRR configuration for North South traffic Spine and PE ?
Might it feature request.
Right now each reload override frr.conf on all nodes.

Thank you.

spirit · Jun 8, 2022

You can keep your own config with creating an /etc/frr/frr.conf.local config on each host,
but it'll override mostly all config.

can you share you frr.config ? what is your custom frr north south traffic ? (can't you use bgp controller feature on top of evpn controlller config ?)

volga629 · Jun 8, 2022

spirit said:
You can keep your own config with creating an /etc/frr/frr.conf.local config on each host,
but it'll override mostly all config.

can you share you frr.config ? what is your custom frr north south traffic ? (can't you use bgp controller feature on top of evpn controlller config ?)

Thank you for reply. Separate BGP instance is not contain all required policies and fields in web ui.

Code:

frr version 8.2.2
frr defaults datacenter
hostname pve00
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
vrf vrf_vmbr2
 vni 10000
exit-vrf
!
router bgp 65000
 bgp router-id 1.1.1.1
 coalesce-time 1000
 neighbor NX3K-LEAF peer-group
 neighbor NX3K-LEAF remote-as 65531
 neighbor NX3K-LEAF graceful-restart
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor  leaf1 peer-group NX3K-LEAF
 neighbor leaf2 peer-group NX3K-LEAF
 neighbor pve02 peer-group VTEP
 neighbor pve03 peer-group VTEP
 neighbor pve04 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor NX3K-LEAF route-map VPN-internet-connected in
  neighbor NX3K-LEAF route-map VPN-internet-connected out
  no neighbor VTEP activate
  rd vpn export 6500:100
  rt vpn import 6500:100 6500:10 6500:12 6500:14 65531:997
  rt vpn export 6500:100 6500:10 6500:12 6500:14
  export vpn
  import vpn
 exit-address-family
 !
 address-family ipv4 vpn
  neighbor NX3K-LEAF activate
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_vmbr2
 bgp router-id 1.1.1.1
 !
 address-family ipv4 unicast
  redistribute connected route-map VMBR2-internet-connected
  rd vpn export 6500:100
  rt vpn import 6500:100 6500:10 6500:12 6500:14 65531:997
  rt vpn export 6500:100
  export vpn
  import vpn
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast
  advertise ipv6 unicast
  route-target import 6500:1000
 exit-address-family
exit
!
access-list acl-nx3k-leaf seq 10 permit 172.24.100.0/28
access-list acl-nx3k-leaf seq 20 permit 10.0.1.0/24
access-list acl-nx3k-leaf seq 25 permit 10.0.2.0/28
access-list acl-nx3k-leaf seq 27 permit 0.0.0.0
access-list acl-vmbr2-leaf seq 5 permit 172.24.100.0/28
access-list acl-vmbr2-leaf seq 20 permit 10.0.1.0/24
access-list acl-vmbr2-leaf seq 25 permit 10.0.2.0/28
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
route-map VPN-internet-connected permit 1001
 match ip address acl-nx3k-leaf
 match ip address prefix-len 32
 set extcommunity rt 6500:10
exit
!
route-map VMBR2-internet-connected permit 1001
 match ip address acl-vmbr2-leaf
 match ip address prefix-len 32
 set extcommunity rt 6500:100
exit
!

spirit · Jun 8, 2022

thanks.

I'll be a little bit complex to add all theses options (I was planning to add route-map && access list in the future).

for now, you can use the frr.conf.local .

(proxmox still generate the begin of the conf up to the vrf section ,just before bgp section, and after merge the config of frr.conf.local

volga629 · Jun 11, 2022

spirit said:
thanks.

I'll be a little bit complex to add all theses options (I was planning to add route-map && access list in the future).

for now, you can use the frr.conf.local .

(proxmox still generate the begin of the conf up to the vrf section ,just before bgp section, and after merge the config of frr.conf.local

But if I add one more network for East West how it going do diff ??? How I can file feature request , that really important feature to implement for SDN solution.

spirit · Jun 12, 2022

volga629 said:
But if I add one more network for East West how it going do diff ???

I don't known exactly how is your network (do you use mpls ? as I'm seeing impot/export vpn).
when you say "add on more network", do you mean external network ?
can you show me a diff if you should do is manually ?

volga629 said:
How I can file feature request , that really important feature to implement for SDN solution.

you can fill a request in bugzilla.proxmox.com.

I have some request in queue to implement ospf in underlay.
currently only bgp is implemented for underlay. ("add bgp controller" in gui)
in your case, it seem to be some kind of vpn / mpls for underlay right ?

volga629 · Jun 13, 2022

spirit said:
I don't known exactly how is your network (do you use mpls ? as I'm seeing impot/export vpn).
when you say "add on more network", do you mean external network ?
can you show me a diff if you should do is manually ?

you can fill a request in bugzilla.proxmox.com.

I have some request in queue to implement ospf in underlay.
currently only bgp is implemented for underlay. ("add bgp controller" in gui)
in your case, it seem to be some kind of vpn / mpls for underlay right ?

Prefer implement ISIS frr (optional , not all networks require underlay ) as underlay if goes integration into L3DCI services, because in my experience OSPF rear is use as underlay. We not use MPLS only EVPN, VXLAN (Spine, Leaf or Cisco ACI third party vendor) . I am asking, because I need keep my configuration portion and configuration from Proxmox some how to be merge into one.
I am not sure if need to monitor frr.conf for changes and then patch it with diff, or will be more cleaner have custom configuration text area which will be merged on commit .

spirit · Jun 14, 2022

yes, I could try to merge configuration sections instead simply override it. I think it shouldn't be too difficult.

For my personal culture,
what is the usage of "rd vpn export /import" in your config ? is is related to ISIS ? (I'm not an network expert, but I try to learn different protocol to implement a nice gui for users )

volga629 · Jun 29, 2022

I will file enchantment request, because I will need start migrate 120 clients from vmware clusters and it constantly override FRR configuration.
Might need some bash script and monitor FRR configuration file for changes as work around.
If you will need my help to test the solution, I will be really glad to help out.

volga629 · Jun 29, 2022

ISIS as underlay and reachability protocol is match more robust then OSPF.
Features like segment routing and traffic engineering allow match more detail network segmentation .
rd vpn export /import is related to North South traffic toward PE where you control what class networks will be advertised in BGP on PE ( class 2 meaning /32 and class 5 meaning subnet ).

spirit · Jun 30, 2022

Hi, I was a bit busy last weeks, I'll try to work on this (merge the configurations) before going to holiday in 2weeks.

I need to code a frr config parser to parse the frr.local configuration, then it should be easy de merge sections. (I'm not sure if ordering of config inside a section is important or not)

Maybe later I could add support for isis in the gui, and access-list/routemap, but it'll take a litte bit more time.

BTW, if you don't declare any new zones (no new evpn vrf), but only new vnets,
I think you could try to put the frr.config in readonly with

chattr +i /etc/frr/frr.conf
(like this, even root process can't change it)

volga629 · Jun 30, 2022

Thank you, frr local configuration will be great to preserve custom configuration. Put into read only not an option, because I need push a lot of VNET 's . For East West zones and BGP is pretty match static.

volga629 · Jun 30, 2022

Here are ticket for feature request.

https://bugzilla.proxmox.com/show_bug.cgi?id=4127

spirit · Jun 30, 2022

volga629 said:
Thank you, frr local configuration will be great to preserve custom configuration. Put into read only not an option, because I need push a lot of VNET 's . For East West zones and BGP is pretty match static.

what is the frr configuration change when you add a new vnet ? (I'm already thinking about the future plugin support

volga629 · Jun 30, 2022

In order test this I need resolve out sync issue. Right now 2 nodes in pending state with message.
And I do see it missing configuration.
I think what missing

1. Peer group definition
2. Access list ( route map )
3. Address family under vrf and global level.

Code:

 local sdn network configuration is too old, please reload

Here are configuration which pushed by the module
By the fay frr version seems hardcoded, right now FRR version 8.2 is running.

Code:

frr version 8.0.1
frr defaults datacenter
hostname pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_vmbr2
 vni 10000
exit-vrf
!
router bgp 65000
 bgp router-id 172.24.100.4
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 172.24.100.5 peer-group VTEP
 neighbor 172.24.100.6 peer-group VTEP
 neighbor 172.24.100.7 peer-group VTEP
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
!
router bgp 65000 vrf vrf_vmbr2
 bgp router-id 172.24.100.4
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast
  advertise ipv6 unicast
  route-target import 6500:1000
 exit-address-family
!
route-map MAP_VTEP_IN permit 1
!
route-map MAP_VTEP_OUT permit 1
!
line vty

spirit · Jul 2, 2022

I'll fix the wrong version number.

one question:

do you really need
"
rd vpn export 6500:100
rt vpn import 6500:100 6500:10 6500:12 6500:14 65531:997
rt vpn export 6500:100 6500:10 6500:12 6500:14
export vpn
"

on both global bgp router && vrf bgp router ?

or does it work with only vrf ? (as vms in a zone, are on vnet with vxlan interface in this vrf).

(I'm asking that if I need to implemented a clean gui, the vrf section options should be done in the zone form)

Also, could you tell me what is the difference in your config when you add/remove a vnet ?

volga629 · Jul 2, 2022

spirit said:
do you really need
"
rd vpn export 6500:100
rt vpn import 6500:100 6500:10 6500:12 6500:14 65531:997
rt vpn export 6500:100 6500:10 6500:12 6500:14
export vpn
"

Yes, we do need rd rt from global or vrf tables ( reason, because we tell BGP what routes we want to see in main table or vrf table. )
When you define access list with route map that where are you can set BGP ext-community or community ( community determine what type of route will be set on peer side)

Test diff

Code:

# cat test.patch
--- /etc/frr/frr.conf    2022-06-29 13:54:35.375705512 -0500
+++ /opt/frr.conf    2022-06-29 13:49:19.662967061 -0500
@@ -1,37 +1,62 @@
-frr version 8.0.1
+frr version 8.2.2
 frr defaults datacenter
 hostname aitdalpve01
 log syslog informational
+no ipv6 forwarding
 service integrated-vtysh-config
 !
-!
 vrf vrf_vmbr2
  vni 10000
 exit-vrf
 !
 router bgp 65000
  bgp router-id 172.24.100.4
- no bgp default ipv4-unicast
  coalesce-time 1000
+ neighbor NX3K-LEAF peer-group
+ neighbor NX3K-LEAF remote-as 65531
+ neighbor NX3K-LEAF graceful-restart
  neighbor VTEP peer-group
  neighbor VTEP remote-as 65000
  neighbor VTEP bfd
+ neighbor 172.24.100.8 peer-group NX3K-LEAF
+ neighbor 172.24.100.9 peer-group NX3K-LEAF
  neighbor 172.24.100.5 peer-group VTEP
  neighbor 172.24.100.6 peer-group VTEP
  neighbor 172.24.100.7 peer-group VTEP
  !
+ address-family ipv4 unicast
+  neighbor NX3K-LEAF route-map VPN-internet-connected in
+  neighbor NX3K-LEAF route-map VPN-internet-connected out
+  no neighbor VTEP activate
+  rd vpn export 6500:100
+  rt vpn import 6500:100 6500:10 6500:12 6500:14 65531:997
+  rt vpn export 6500:100 6500:10 6500:12 6500:14
+  export vpn
+  import vpn
+ exit-address-family
+ !
+ address-family ipv4 vpn
+  neighbor NX3K-LEAF activate
+ exit-address-family
+ !
  address-family l2vpn evpn
+  neighbor VTEP activate
   neighbor VTEP route-map MAP_VTEP_IN in
   neighbor VTEP route-map MAP_VTEP_OUT out
-  neighbor VTEP activate
   advertise-all-vni
  exit-address-family
+exit
 !
 router bgp 65000 vrf vrf_vmbr2
  bgp router-id 172.24.100.4
  !
  address-family ipv4 unicast
-  redistribute connected
+  redistribute connected route-map VMBR2-internet-connected
+  rd vpn export 6500:100
+  rt vpn import 6500:100 6500:10 6500:12 6500:14 65531:997
+  rt vpn export 6500:100
+  export vpn
+  import vpn
  exit-address-family
  !
  address-family ipv6 unicast
@@ -43,10 +68,31 @@
   advertise ipv6 unicast
   route-target import 6500:1000
  exit-address-family
+exit
+!
+access-list acl-nx3k-leaf seq 10 permit 172.24.100.0/28
+access-list acl-nx3k-leaf seq 20 permit 10.0.1.0/24
+access-list acl-nx3k-leaf seq 25 permit 10.0.2.0/28
+access-list acl-nx3k-leaf seq 27 permit 0.0.0.0
+access-list acl-vmbr2-leaf seq 5 permit 172.24.100.0/28
+access-list acl-vmbr2-leaf seq 20 permit 10.0.1.0/24
+access-list acl-vmbr2-leaf seq 25 permit 10.0.2.0/28
 !
 route-map MAP_VTEP_IN permit 1
+exit
 !
 route-map MAP_VTEP_OUT permit 1
+exit
+!
+route-map VPN-internet-connected permit 1001
+ match ip address acl-nx3k-leaf
+ match ip address prefix-len 32
+ set extcommunity rt 6500:10
+exit
+!
+route-map VMBR2-internet-connected permit 1001
+ match ip address acl-vmbr2-leaf
+ match ip address prefix-len 32
+ set extcommunity rt 6500:100
+exit
 !
-line vty
-!
\ No newline at end of file

spirit · Jul 12, 2022

Hi,
here a patched version to merge the frr local config

Code:

wget https://mutulin1.odiso.net/libpve-network-perl_0.7.1_all.deb
dpkg -i libpve-network-perl_0.7.1_all.deb

Currently it's only merge && append in different sections,
so

Code:

+  redistribute connected
+  redistribute connected route-map VMBR2-internet-connected

It'll be duplicated. (but redistributed connected is only added if you use the "advertise subnet" option or if you define an exit-node)

Could you already test it ?

I'll try to polish it before going to holiday next week.

volga629 · Jul 13, 2022

spirit said:
Hi,
here a patched version to merge the frr local config

Code:

wget https://mutulin1.odiso.net/libpve-network-perl_0.7.1_all.deb dpkg -i libpve-network-perl_0.7.1_all.deb

Currently it's only merge && append in different sections,
so

Code:

+ redistribute connected + redistribute connected route-map VMBR2-internet-connected

It'll be duplicated. (but redistributed connected is only added if you use the "advertise subnet" option or if you define an exit-node)

Could you already test it ?

I'll try to polish it before going to holiday next week.

Thank you for release, I will test soon
If I am going put frr.local my config file as is what is going to happened ?

SDN FRR custom configuration

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Member

Distinguished Member

Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member