Integrating Proxmox SDN with existing SDN network

mbosma · Sep 12, 2023

spirit said:
ok, fixed, it's was a small bug on "node" param not parsed, then the isis config was not loaded and frr config not generated.

can you try it again (same link)
libpve-network-perl_0.8.1_all.deb 9e318acae4455e4339980d1099e09074

This one works as intended! Was able to have 2 isis sessions and have them failover:

Code:

pve4-test# show isis interface
Area 1:
  Interface   CircId   State    Type     Level
  eno7        0x0      Up       lan      L1L2     
  eno8        0x7      Up       lan      L1L2

Code:

pve4-test# show isis route    
Area 1:
IS-IS L1 IPv4 routing table:

 Prefix            Metric  Interface  Nexthop   Label(s)   
 ---------------------------------------------------------
 10.0.0.0/31       20      eno8       10.1.0.6  -          
 10.0.0.2/31       20      eno7       10.1.0.8  -          
 10.1.0.0/31       20      eno8       10.1.0.6  -          
 10.1.0.2/31       20      eno7       10.1.0.8  -          
 10.1.0.4/31       20      eno7       10.1.0.8  -          
 10.1.0.6/31       20      eno8       10.1.0.6  -          
 10.1.0.8/31       20      eno7       10.1.0.8  -          
 10.66.66.0/24     20      eno7       10.1.0.8  -          
 10.99.99.1/32     20      eno7       10.1.0.8  -          
                           eno8       10.1.0.6  -          
 10.99.99.2/32     10      eno8       10.1.0.6  -          
 10.99.99.3/32     10      eno7       10.1.0.8  -          
 10.99.99.4/32     20      eno8       10.1.0.6  -          
 10.99.99.5/32     20      eno7       10.1.0.8  -          
 10.99.99.6/32     20      eno7       10.1.0.8  -          
 10.99.99.7/32     0       -          -         -          
 192.168.245.0/24  20      eno7       10.1.0.8  -          
                           eno8       10.1.0.6  -          

IS-IS L2 IPv4 routing table:

 Prefix            Metric  Interface  Nexthop   Label(s)   
 ---------------------------------------------------------
 0.0.0.0/0         20      eno7       10.1.0.8  -          
                           eno8       10.1.0.6  -          
 10.0.0.0/31       20      eno8       10.1.0.6  -          
 10.0.0.2/31       20      eno7       10.1.0.8  -          
 10.1.0.0/31       20      eno8       10.1.0.6  -          
 10.1.0.2/31       20      eno7       10.1.0.8  -          
 10.1.0.4/31       20      eno7       10.1.0.8  -          
 10.1.0.6/31       20      eno8       10.1.0.6  -          
 10.1.0.8/31       20      eno7       10.1.0.8  -          
 10.66.66.0/24     20      eno7       10.1.0.8  -          
 10.99.99.1/32     20      eno7       10.1.0.8  -          
                           eno8       10.1.0.6  -          
 10.99.99.2/32     10      eno8       10.1.0.6  -          
 10.99.99.3/32     10      eno7       10.1.0.8  -          
 10.99.99.4/32     20      eno8       10.1.0.6  -          
 10.99.99.5/32     20      eno7       10.1.0.8  -          
 10.99.99.6/32     20      eno7       10.1.0.8  -          
 10.99.99.7/32     20      eno7       10.1.0.8  -          
                           eno8       10.1.0.6  -          
 192.168.245.0/24  20      eno7       10.1.0.8  -          
                           eno8       10.1.0.6  -

We ran into a small issue because of the interface ip-addresses being used as VTEP addresses causing downtime.
This is fixed by using the same workaround in the interfaces file from this thread:
https://forum.proxmox.com/threads/s...onfiguration-to-use-it-over-wiregaurd.106401/

Code:

iface vxlan_vni100
        vxlan-local-tunnelip 10.99.99.7

This makes sure the VTEP ip is set to the ip-address on the loopback interface which is available on all links:

Code:

pve4-test# show evpn vni detail
VNI: 100
 Type: L2
 Tenant VRF: vrf_evpnz1
 VxLAN interface: vxlan_vni100
 VxLAN ifIndex: 63
 SVI interface: vni100
 SVI ifIndex: 12
 Local VTEP IP: 10.99.99.7

It would be nice to have that available in the interface as well but this is definitely workable for us.

spirit said:
I'll keep you in touch, thanks for your patience. (Sorry to be late, I was very busy at work this week)

Thank you so much for your work! Not at all a problem, I'm already glad you're willing to help.

Once these changed make it into the repo we'll start using it in our new cluster.

If you have any questions regarding this setup to get it ready for the repo please let me know.

spirit · Sep 12, 2023

mbosma said:

This one works as intended! Was able to have 2 isis sessions and have them failover:

Code:

pve4-test# show isis interface
Area 1:
  Interface   CircId   State    Type     Level
  eno7        0x0      Up       lan      L1L2   
  eno8        0x7      Up       lan      L1L2

Code:

pve4-test# show isis route  
Area 1:
IS-IS L1 IPv4 routing table:

 Prefix            Metric  Interface  Nexthop   Label(s) 
 ---------------------------------------------------------
 10.0.0.0/31       20      eno8       10.1.0.6  -        
 10.0.0.2/31       20      eno7       10.1.0.8  -        
 10.1.0.0/31       20      eno8       10.1.0.6  -        
 10.1.0.2/31       20      eno7       10.1.0.8  -        
 10.1.0.4/31       20      eno7       10.1.0.8  -        
 10.1.0.6/31       20      eno8       10.1.0.6  -        
 10.1.0.8/31       20      eno7       10.1.0.8  -        
 10.66.66.0/24     20      eno7       10.1.0.8  -        
 10.99.99.1/32     20      eno7       10.1.0.8  -        
                           eno8       10.1.0.6  -        
 10.99.99.2/32     10      eno8       10.1.0.6  -        
 10.99.99.3/32     10      eno7       10.1.0.8  -        
 10.99.99.4/32     20      eno8       10.1.0.6  -        
 10.99.99.5/32     20      eno7       10.1.0.8  -        
 10.99.99.6/32     20      eno7       10.1.0.8  -        
 10.99.99.7/32     0       -          -         -        
 192.168.245.0/24  20      eno7       10.1.0.8  -        
                           eno8       10.1.0.6  -        

IS-IS L2 IPv4 routing table:

 Prefix            Metric  Interface  Nexthop   Label(s) 
 ---------------------------------------------------------
 0.0.0.0/0         20      eno7       10.1.0.8  -        
                           eno8       10.1.0.6  -        
 10.0.0.0/31       20      eno8       10.1.0.6  -        
 10.0.0.2/31       20      eno7       10.1.0.8  -        
 10.1.0.0/31       20      eno8       10.1.0.6  -        
 10.1.0.2/31       20      eno7       10.1.0.8  -        
 10.1.0.4/31       20      eno7       10.1.0.8  -        
 10.1.0.6/31       20      eno8       10.1.0.6  -        
 10.1.0.8/31       20      eno7       10.1.0.8  -        
 10.66.66.0/24     20      eno7       10.1.0.8  -        
 10.99.99.1/32     20      eno7       10.1.0.8  -        
                           eno8       10.1.0.6  -        
 10.99.99.2/32     10      eno8       10.1.0.6  -        
 10.99.99.3/32     10      eno7       10.1.0.8  -        
 10.99.99.4/32     20      eno8       10.1.0.6  -        
 10.99.99.5/32     20      eno7       10.1.0.8  -        
 10.99.99.6/32     20      eno7       10.1.0.8  -        
 10.99.99.7/32     20      eno7       10.1.0.8  -        
                           eno8       10.1.0.6  -        
 192.168.245.0/24  20      eno7       10.1.0.8  -        
                           eno8       10.1.0.6  -

We ran into a small issue because of the interface ip-addresses being used as VTEP addresses causing downtime.
This is fixed by using the same workaround in the interfaces file from this thread:
https://forum.proxmox.com/threads/s...onfiguration-to-use-it-over-wiregaurd.106401/

Code:

iface vxlan_vni100
        vxlan-local-tunnelip 10.99.99.7

This makes sure the VTEP ip is set to the ip-address on the loopback interface which is available on all links:

Code:

pve4-test# show evpn vni detail
VNI: 100
 Type: L2
 Tenant VRF: vrf_evpnz1
 VxLAN interface: vxlan_vni100
 VxLAN ifIndex: 63
 SVI interface: vni100
 SVI ifIndex: 12
 Local VTEP IP: 10.99.99.7

It would be nice to have that available in the interface as well but this is definitely workable for us.

Thank you so much for your work! Not at all a problem, I'm already glad you're willing to help.

Once these changed make it into the repo we'll start using it in our new cluster.

If you have any questions regarding this setup to get it ready for the repo please let me know.

can you share your /etc/network/interfaces with loopback config ?

I already have implement the loopback option for the bgp controller, so it's easy to port it for isis too.

spirit · Sep 12, 2023

ok, I didn't see your loopback in previous /etc/network/interfaces post.

can you download && install the deb again (same link)

c14662f8983035e508f989967f7488aa libpve-network-perl_0.8.1_all.deb

option in controllers.cfg:

Code:

   isis-ifaces:   eth0,eth1,eth2
   isis-domain:  yourdomain
   isis-net: 10.0000.0000.0005.00
   loopback: lo1
   node: yournode

This will update vxlan-local-tunnelip with loopback iface ip, but also frr.conf for bgp routerid[/CODE]

mbosma · Sep 13, 2023

spirit said:
ok, I didn't see your loopback in previous /etc/network/interfaces post.

can you download && install the deb again (same link)

c14662f8983035e508f989967f7488aa libpve-network-perl_0.8.1_all.deb

option in controllers.cfg:

Code:

isis-ifaces: eth0,eth1,eth2 isis-domain: yourdomain isis-net: 10.0000.0000.0005.00 loopback: lo1 node: yournode

This will update vxlan-local-tunnelip with loopback iface ip, but also frr.conf for bgp routerid[/CODE]

Thank you for this update!
It's almost working as intended but assigns the wrong ip-address.

Setting controllers.cfg loopback to lo

Code:

isis: pve4-test
        isis-domain 1
        isis-ifaces eno8,eno7
        isis-net 10.0000.0000.0007.00
        loopback: lo
        node pve4-test

And having the following in interfaces:

Code:

auto lo
iface lo inet static
        address 10.99.99.7/32

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.245.204/24
        gateway 192.168.245.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto eno7
iface eno7 inet static
        address 10.1.0.9/31
        mtu 9000

auto eno8
iface eno8 inet static
        address 10.1.0.7/31
        mtu 9000

Stil results in the server using 10.1.0.7 as loopback (eno8 is the primary interface in this case).

Code:

auto vxlan_vni100
iface vxlan_vni100
        vxlan-id 100
        vxlan-local-tunnelip 10.1.0.7
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 9000

The same goes for frr.conf, I can see it's adding bgp router-id now but with the wrong address:

Code:

frr version 8.5.1
frr defaults datacenter
hostname pve4-test
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpnz1
 vni 1001
exit-vrf
!
interface eno7
 ip router isis 1
!
interface eno8
 ip router isis 1
!
router bgp 65000
 bgp router-id 10.1.0.7
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 10.99.99.1 peer-group VTEP
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_evpnz1
 bgp router-id 10.1.0.7
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast
  advertise ipv6 unicast
 exit-address-family
exit
!
router isis 1
 net 10.0000.0000.0007.00
 redistribute ipv4 connected level-1
 redistribute ipv6 connected level-1
 log-adjacency-changes
exit
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
line vty
!

Did I do something wrong in the controllers.cfg?

spirit · Sep 13, 2023

typo error ? (the : on the loopback )

Code:

isis: pve4-test
        isis-domain 1
        isis-ifaces eno8,eno7
        isis-net 10.0000.0000.0007.00
        loopback: lo
        node pve4-test

-->

Code:

isis: pve4-test
        isis-domain 1
        isis-ifaces eno8,eno7
        isis-net 10.0000.0000.0007.00
        loopback lo
        node pve4-test

mbosma · Sep 13, 2023

spirit said:

typo error ? (the : on the loopback )

Code:

isis: pve4-test
        isis-domain 1
        isis-ifaces eno8,eno7
        isis-net 10.0000.0000.0007.00
        loopback: lo
        node pve4-test

-->

Code:

isis: pve4-test
        isis-domain 1
        isis-ifaces eno8,eno7
        isis-net 10.0000.0000.0007.00
        loopback lo
        node pve4-test

Yes that works perfectly! Stupid mistake from my side.
All servers now use their loopback address.

spirit · Sep 13, 2023

mbosma said:
Yes that works perfectly! Stupid mistake from my side.
All servers now use their loopback address.

ok thanks! I'll send patch to the pve-devel mailing list. (with gui && docs too)

mbosma · Oct 13, 2023

spirit said:
ok thanks! I'll send patch to the pve-devel mailing list. (with gui && docs too)

I can see your patch in the mailing list, would it help if I created a feature request in git?

hvisage · Mar 30, 2024

@spirit now for related IPv6 (and perhaps openfabric?) parsing of the interfaces too?
Perhaps enhancing the interface to know/enable IPv6?

https://bugzilla.proxmox.com/show_bug.cgi?id=5343

root@red:/etc/frr# cat frr.conf
frr version 8.5.2
frr defaults datacenter
hostname red
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_MenMevpn
vni 1
exit-vrf
!
interface eno1
ip router isis menm
!
interface eno2
ip router isis menm
!
router bgp 65000
bgp router-id 2c0f:c40:feed:feed:1:0:5:1
no bgp hard-administrative-reset
no bgp default ipv4-unicast
coalesce-time 1000
no bgp graceful-restart notification
neighbor VTEP peer-group
neighbor VTEP remote-as 65000
neighbor VTEP bfd
neighbor VTEP update-source 10.5.127.1
neighbor 2c0f:c40:feed:feed:1:0:5:2 peer-group VTEP
neighbor 2c0f:c40:feed:feed:1:0:5:3 peer-group VTEP
bgp bestpath as-path multipath-relax
neighbor BGP peer-group
neighbor BGP remote-as 65000
neighbor BGP bfd
neighbor 10.5.127.2 peer-group BGP
neighbor 10.5.127.3 peer-group BGP
!
address-family ipv4 unicast
network 102.212.60.251/32
neighbor BGP activate
neighbor BGP soft-reconfiguration inbound
import vrf vrf_MenMevpn
exit-address-family
!
address-family ipv6 unicast
import vrf vrf_MenMevpn
exit-address-family
!
address-family l2vpn evpn
neighbor VTEP activate
neighbor VTEP route-map MAP_VTEP_IN in
neighbor VTEP route-map MAP_VTEP_OUT out
advertise-all-vni
exit-address-family
exit
!
router bgp 65000 vrf vrf_MenMevpn
bgp router-id 2c0f:c40:feed:feed:1:0:5:1
no bgp hard-administrative-reset
no bgp graceful-restart notification
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!
address-family ipv6 unicast
redistribute connected
exit-address-family
!
address-family l2vpn evpn
default-originate ipv4
default-originate ipv6
exit-address-family
exit
!
router isis menm
net menm
redistribute ipv4 connected level-1
redistribute ipv6 connected level-1
log-adjacency-changes
exit
!
ip prefix-list loopbacks_ips seq 10 permit 0.0.0.0/0 le 32
ip prefix-list only_default seq 1 permit 0.0.0.0/0
!
ipv6 prefix-list only_default_v6 seq 1 permit ::/0
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
match ip address prefix-list only_default
set metric 200
exit
!
route-map MAP_VTEP_OUT permit 2
match ipv6 address prefix-list only_default_v6
set metric 200
exit
!
route-map MAP_VTEP_OUT permit 3
exit
!
route-map correct_src permit 1
match ip address prefix-list loopbacks_ips
set src 102.212.60.251
exit
!
ip protocol bgp route-map correct_src
!
line vty

!root@red:/etc/frr# cat b/frr.conf
frr version 8.5.2
frr defaults datacenter
hostname red
log syslog informational
service integrated-vtysh-config
!
interface eno2
description green
ipv6 router isis menm
exit
!
interface eno1
ip router isis menm
ipv6 router isis menm
ipv6 router openfabric menm
exit
!
interface coro1
ipv6 router isis menm
exit
!
interface coro2
ipv6 router isis menm
exit
!
router bgp 65000
bgp router-id 102.212.60.251
no bgp hard-administrative-reset
no bgp default ipv4-unicast
coalesce-time 1000
no bgp graceful-restart notification
bgp bestpath as-path multipath-relax
neighbor BGP peer-group
neighbor BGP remote-as 65000
neighbor BGP bfd
neighbor VTEP peer-group
neighbor VTEP remote-as 65000
neighbor VTEP bfd
neighbor VTEP update-source 10.5.127.1
neighbor 10.5.127.1 peer-group VTEP
neighbor 10.5.127.2 peer-group VTEP
neighbor 10.5.127.3 peer-group VTEP
!
address-family ipv4 unicast
network 102.212.60.251/32
neighbor BGP activate
neighbor BGP soft-reconfiguration inbound
exit-address-family
!
address-family l2vpn evpn
neighbor VTEP activate
neighbor VTEP route-map MAP_VTEP_IN in
neighbor VTEP route-map MAP_VTEP_OUT out
advertise-all-vni
exit-address-family
exit
!
router isis menm
is-type level-1
net 49.0001.1111.1111.1111.00
redistribute ipv4 connected level-1
redistribute ipv6 connected level-1
topology ipv6-unicast
log-adjacency-changes
exit
!
ip prefix-list loopbacks_ips seq 10 permit 0.0.0.0/0 le 32
!
route-map MAP_VTEP_IN permit 1
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
route-map correct_src permit 1
match ip address prefix-list loopbacks_ips
set src 102.212.60.251
exit
!
ipv6 protocol connected route-map test
!
ip protocol bgp route-map correct_src
!
router openfabric menm
exit
!

hvisage · Mar 30, 2024

oh, and there are another "bug" in the parsing/do of the Network ID of the ISIS plugin that fails FRR parsing as the GUI/plugin doesn't parse that before hand and doesn't report that error in the task output ;(

Search

Search

Integrating Proxmox SDN with existing SDN network

mbosma

Renowned Member

spirit

Distinguished Member

spirit

Distinguished Member

mbosma

Renowned Member

spirit

Distinguished Member

mbosma

Renowned Member

spirit

Distinguished Member

mbosma

Renowned Member

hvisage

Renowned Member

hvisage

Renowned Member

We value your privacy