I've been trying to integrate the Proxmox SDN into an existing vxlan network using IS-IS.
This way we'll be able to use the different vnets across multiple clusters as well as bind that to a vlan to attach legacy devices.
Our lab setup is using a route-reflector on a spine switch and 2 leaf switches.
Using IS-IS as underlay and evpn the proxmox servers (2 in a cluster and 1 separate) are able to receive routes from the route-reflector.
This way vm's can communicate with other vm's in the vnet as well as vm's on other clusters and devices connected to that vni.
This lab setup turned out sort of successfully but here's the catch.
There are a couple of options the Proxmox SDN plugin pushes (and doesn't push) which requires manual steps each time the config reloads.
A couple of changes I can configure in frr.conf.local:
The isis router:
And:
For some reason frr doesn't want to act as route-reflector client when these settings are applied even though permit is set to 1.
I'm no frr expert by all means but removing these worked.
The following changes however are not parsed in frr.conf.local and is actually the most vital one.
Without this change the servers lose the IS-IS underlay and all routes essentially bringing down the whole network on the hosts.
I tried a bit of hacking in the SDN plugin but my perl is not up to the task.
Would it be possible to parse the interface codeblocks?
Here's some more information for those interested:
A diagram of the connected devices:
These are are the SDN configs:
And network config:
And the working running config of frr:
I hope the output of my journey might help someone stumbling upon this topic in the future.
If a dev could look into my request regarding parsing the interface block in frr.conf.local that would be greatly appreciated.
This way we'll be able to use the different vnets across multiple clusters as well as bind that to a vlan to attach legacy devices.
Our lab setup is using a route-reflector on a spine switch and 2 leaf switches.
Using IS-IS as underlay and evpn the proxmox servers (2 in a cluster and 1 separate) are able to receive routes from the route-reflector.
This way vm's can communicate with other vm's in the vnet as well as vm's on other clusters and devices connected to that vni.
This lab setup turned out sort of successfully but here's the catch.
There are a couple of options the Proxmox SDN plugin pushes (and doesn't push) which requires manual steps each time the config reloads.
A couple of changes I can configure in frr.conf.local:
The isis router:
Code:
router isis 1
net 10.0000.0000.0005.00
redistribute ipv4 connected level-1
log-adjacency-changes
exitAnd:
Code:
router bgp 65000
address-family l2vpn evpn
no neighbor VTEP route-map MAP_VTEP_IN in
no neighbor VTEP route-map MAP_VTEP_OUT out
exit
exitI'm no frr expert by all means but removing these worked.
The following changes however are not parsed in frr.conf.local and is actually the most vital one.
Code:
interface vmbr1
ip router isis 1
exitI tried a bit of hacking in the SDN plugin but my perl is not up to the task.
Would it be possible to parse the interface codeblocks?
Here's some more information for those interested:
A diagram of the connected devices:
These are are the SDN configs:
Code:
root@pve1-test:~# cat /etc/pve/sdn/controllers.cfg
evpn: evpn
asn 65000
peers 10.99.99.1
root@pve1-test:~# cat /etc/pve/sdn/zones.cfg
evpn: evpnz1
controller evpn
vrf-vxlan 1001
advertise-subnets 1
ipam pve
mac DE:AA:11:CD:D5:98
mtu 9000
root@pve1-test:~# cat /etc/pve/sdn/vnets.cfg
vnet: vni100
zone evpnz1
tag 100
vnet: vni300
zone evpnz1
tag 300
vnet: vni200
zone evpnz1
tag 200And network config:
Code:
auto lo
iface lo inet static
address 10.99.99.4/32
iface lo inet static
address 10.99.99.4/32
iface eno1 inet manual
iface eno2 inet manual
iface enp8s0f0 inet manual
auto enp8s0f1
iface enp8s0f1 inet manual
mtu 9000
auto vmbr0
iface vmbr0 inet static
address 192.168.245.201/24
gateway 192.168.245.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet static
address 10.1.0.1/31
bridge-ports enp8s0f1
bridge-stp off
bridge-fd 0
mtu 9000
auto vmbr2
iface vmbr2 inet static
address 10.66.100.20/24
bridge-ports enp8s0f0
bridge-stp off
bridge-fd 0
source /etc/network/interfaces.d/*And the working running config of frr:
Code:
!
frr version 8.5.1
frr defaults datacenter
hostname pve1-test
log syslog informational
service integrated-vtysh-config
!
vrf vrf_evpnz1
vni 1001
exit-vrf
!
interface vmbr1
ip router isis 1
exit
!
router bgp 65000
bgp router-id 10.1.0.1
no bgp hard-administrative-reset
no bgp default ipv4-unicast
coalesce-time 1000
no bgp graceful-restart notification
neighbor VTEP peer-group
neighbor VTEP remote-as 65000
neighbor VTEP bfd
neighbor 10.99.99.1 peer-group VTEP
!
address-family l2vpn evpn
neighbor VTEP activate
advertise-all-vni
exit-address-family
exit
!
router bgp 65000 vrf vrf_evpnz1
bgp router-id 10.1.0.1
no bgp hard-administrative-reset
no bgp graceful-restart notification
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!
address-family ipv6 unicast
redistribute connected
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
advertise ipv6 unicast
exit-address-family
exit
!
router isis 1
net 10.0000.0000.0004.00
redistribute ipv4 connected level-1
log-adjacency-changes
exit
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
endI hope the output of my journey might help someone stumbling upon this topic in the future.
If a dev could look into my request regarding parsing the interface block in frr.conf.local that would be greatly appreciated.
