proxmox 7.0 sdn beta test

Ruffy91

Member
May 23, 2017
10
0
21
31
Enjoy your holiday :)

The config as generated now is:
Code:
frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_evpn
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

I would change it to:
Code:
frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

After I do this a session to my eBGP peer is no longer established.
Unfortunately I am not as fluent with routing and am not sure what I am doing, as a consequence there are probably some errors in there.
 

spirit

Famous Member
Apr 2, 2010
5,780
675
133
www.odiso.com
Enjoy your holiday :)

The config as generated now is:
Code:
frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_evpn
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

I would change it to:
Code:
frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

After I do this a session to my eBGP peer is no longer established.
Unfortunately I am not as fluent with routing and am not sure what I am doing, as a consequence there are probably some errors in there.
Enjoy your holiday :)

The config as generated now is:
Code:
frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_evpn
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

I would change it to:
Code:
frr version 8.0.1
frr defaults datacenter
hostname chsfl1-cl01-pve01
log syslog informational
service integrated-vtysh-config
!
!
vrf vrf_evpn
 vni 500
exit-vrf
!
router bgp 65002
 bgp router-id 100.111.64.3
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor 192.168.102.102 peer-group VTEP
 neighbor 192.168.102.103 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  neighbor VTEP activate
  advertise-all-vni
  autort as 65010
 exit-address-family
!
router bgp 65002 vrf vrf_evpn
 bgp router-id 192.168.102.101
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 4
 neighbor 100.111.64.1 peer-group BGP
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65010:500
  route-target export 65010:500
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
route-map MAP_VTEP_IN deny 1
 match evpn route-type prefix
!
route-map MAP_VTEP_IN permit 2
!
route-map MAP_VTEP_OUT permit 1
!
line vty

After I do this a session to my eBGP peer is no longer established.
Unfortunately I am not as fluent with routing and am not sure what I am doing, as a consequence there are probably some errors in there.
Maybe copy "neighbor bgp activate" in vrf router ipv4 unicast section. But keep it too in main section. I ll do test next week.
 

Ruffy91

Member
May 23, 2017
10
0
21
31
I added a PBR for any traffic on interface evpn01 which added a rule and route table which seem correct, but the rule is ignored by the route selection:
root@chsfl1-cl01-pve01:~# ip rule
300: from all iif evpn01 lookup 10000 proto zebra
1000: from all lookup [l3mdev-table]
32765: from all lookup local
32766: from all lookup main
32767: from all lookup default
root@chsfl1-cl01-pve01:~# ip route list table 10000
default nhid 673 via 100.111.64.1 dev vmbr0_164 proto pbr metric 20

root@chsfl1-cl01-pve01:~# ip route get 1.1.1.1 from 10.182.3.100 iif evpn01
1.1.1.1 from 10.182.3.100 via 10.182.2.1 dev vmbr0_182
cache iif evpn01

I think it should even be possible to select the correct vrf with PBR (set vrf VRF-NAME) but this wont help if the rules are ignored.
 

spirit

Famous Member
Apr 2, 2010
5,780
675
133
www.odiso.com
I added a PBR for any traffic on interface evpn01 which added a rule and route table which seem correct, but the rule is ignored by the route selection:
root@chsfl1-cl01-pve01:~# ip rule
300: from all iif evpn01 lookup 10000 proto zebra
1000: from all lookup [l3mdev-table]
32765: from all lookup local
32766: from all lookup main
32767: from all lookup default
root@chsfl1-cl01-pve01:~# ip route list table 10000
default nhid 673 via 100.111.64.1 dev vmbr0_164 proto pbr metric 20

root@chsfl1-cl01-pve01:~# ip route get 1.1.1.1 from 10.182.3.100 iif evpn01
1.1.1.1 from 10.182.3.100 via 10.182.2.1 dev vmbr0_182
cache iif evpn01

I think it should even be possible to select the correct vrf with PBR (set vrf VRF-NAME) but this wont help if the rules are ignored.
thanks for testing. I'll do a lab next week when I'll be back at work.
 

grimsrue

New Member
Jul 29, 2022
3
1
3
I wanted to ask if there is going to be an update to allow IPv6 ebgp connectivity?
As it stands now, trying to add IPv6 peering addresses to the the "Peers" window in the SDN UI will add them to the "BGP" peer group. This will mix the v4 and v6 addresses together and wont allow the IPv6 addresses to establish a peering connection. It would be nice to have a v4 and a separate v6 Peer window in the SDN BGP UI so that v4 peering IPs get their own peer group and v6 peering IPs get their own peer group. The IPv6 peer group will also have to be activated under the "address-family ipv6 unicast"

Thinking that it might look something like this below.
Current configuration:

!
frr version 8.3
frr defaults datacenter
hostname PROX-FRR-NSX-B1
log syslog informational
hostname Prox-FrrRouter-B1
service integrated-vtysh-config
!
vrf vrf_EVPNZone
vni 4000
exit-vrf
!
router bgp 48000
bgp router-id 135.xxx.xxx.14
no bgp hard-administrative-reset
no bgp default ipv4-unicast
coalesce-time 1000
no bgp graceful-restart notification
neighbor BGPv4 peer-group
neighbor BGPv4 remote-as external
neighbor BGPv6 peer-group
neighbor BGPv6 remote-as external
neighbor VTEP peer-group
neighbor VTEP remote-as 48000
neighbor VTEP bfd
neighbor 135.xxx.xxx.8 peer-group BGPv4
neighbor 135.xxx.xxx.9 peer-group BGPv4
neighbor 2001:xxxx:xxxx:xxxx::x:4a peer-group BGPv6
neighbor 2001:xxxx:xxxx:xxxx::x:4b peer-group BGPv6
neighbor 135.xxx.xxx.15 peer-group VTEP
neighbor 135.xxx.xxx.16 peer-group VTEP
neighbor 135.xxx.xxx.17 peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:5b peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:6a peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:6b peer-group VTEP
!
address-family ipv4 unicast
neighbor BGPv4 activate
neighbor BGPv4 soft-reconfiguration inbound
import vrf vrf_EVPNZone
exit-address-family
!
address-family ipv6 unicast
neighbor BGPv6 activate
neighbor BGPv6 soft-reconfiguration inbound
import vrf vrf_EVPNZone
exit-address-family
!
address-family l2vpn evpn
neighbor VTEP activate
neighbor VTEP route-map MAP_VTEP_IN in
neighbor VTEP route-map MAP_VTEP_OUT out
advertise-all-vni
exit-address-family
exit
!
router bgp 48000 vrf vrf_EVPNZone
bgp router-id 135.xxx.xxx.14
no bgp hard-administrative-reset
no bgp graceful-restart notification
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!
address-family ipv6 unicast
redistribute connected
exit-address-family
!
address-family l2vpn evpn
default-originate ipv4
default-originate ipv6
exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
end
 

spirit

Famous Member
Apr 2, 2010
5,780
675
133
www.odiso.com
I wanted to ask if there is going to be an update to allow IPv6 ebgp connectivity?
As it stands now, trying to add IPv6 peering addresses to the the "Peers" window in the SDN UI will add them to the "BGP" peer group. This will mix the v4 and v6 addresses together and wont allow the IPv6 addresses to establish a peering connection. It would be nice to have a v4 and a separate v6 Peer window in the SDN BGP UI so that v4 peering IPs get their own peer group and v6 peering IPs get their own peer group. The IPv6 peer group will also have to be activated under the "address-family ipv6 unicast"

Thinking that it might look something like this below.
Current configuration:

!
frr version 8.3
frr defaults datacenter
hostname PROX-FRR-NSX-B1
log syslog informational
hostname Prox-FrrRouter-B1
service integrated-vtysh-config
!
vrf vrf_EVPNZone
vni 4000
exit-vrf
!
router bgp 48000
bgp router-id 135.xxx.xxx.14
no bgp hard-administrative-reset
no bgp default ipv4-unicast
coalesce-time 1000
no bgp graceful-restart notification
neighbor BGPv4 peer-group
neighbor BGPv4 remote-as external
neighbor BGPv6 peer-group
neighbor BGPv6 remote-as external
neighbor VTEP peer-group
neighbor VTEP remote-as 48000
neighbor VTEP bfd
neighbor 135.xxx.xxx.8 peer-group BGPv4
neighbor 135.xxx.xxx.9 peer-group BGPv4
neighbor 2001:xxxx:xxxx:xxxx::x:4a peer-group BGPv6
neighbor 2001:xxxx:xxxx:xxxx::x:4b peer-group BGPv6
neighbor 135.xxx.xxx.15 peer-group VTEP
neighbor 135.xxx.xxx.16 peer-group VTEP
neighbor 135.xxx.xxx.17 peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:5b peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:6a peer-group VTEP
neighbor 2001:xxxx:xxxx:xxxx::x:6b peer-group VTEP
!
address-family ipv4 unicast
neighbor BGPv4 activate
neighbor BGPv4 soft-reconfiguration inbound
import vrf vrf_EVPNZone
exit-address-family
!
address-family ipv6 unicast
neighbor BGPv6 activate
neighbor BGPv6 soft-reconfiguration inbound
import vrf vrf_EVPNZone
exit-address-family
!
address-family l2vpn evpn
neighbor VTEP activate
neighbor VTEP route-map MAP_VTEP_IN in
neighbor VTEP route-map MAP_VTEP_OUT out
advertise-all-vni
exit-address-family
exit
!
router bgp 48000 vrf vrf_EVPNZone
bgp router-id 135.xxx.xxx.14
no bgp hard-administrative-reset
no bgp graceful-restart notification
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!
address-family ipv6 unicast
redistribute connected
exit-address-family
!
address-family l2vpn evpn
default-originate ipv4
default-originate ipv6
exit-address-family
exit
!
route-map MAP_VTEP_IN deny 1
match evpn route-type prefix
exit
!
route-map MAP_VTEP_IN permit 2
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
end
Hi,
yes, I could adapt code for handling mixing ipv4 && ipv6 peer. (I didn't have thinked about this).
Should be easy to implement, I'll try to send a patched version for testing today or tomorrow.
 

spirit

Famous Member
Apr 2, 2010
5,780
675
133
www.odiso.com
@grimsrue

can you test this patched version:

Code:
wget https://mutulin1.odiso.net/libpve-network-perl_0.7.1_all.deb
dpkg -i libpve-network-perl_0.7.1_all.deb

try with mixed ipv4/ipv6 peers , only ipv4 , only ipv6 if possible.
 

grimsrue

New Member
Jul 29, 2022
3
1
3
@grimsrue

can you test this patched version:

Code:
wget https://mutulin1.odiso.net/libpve-network-perl_0.7.1_all.deb
dpkg -i libpve-network-perl_0.7.1_all.deb

try with mixed ipv4/ipv6 peers , only ipv4 , only ipv6 if possible.

Sure! Give me a day or so. Have a busy day today. I'll try to find 30 mins today or tonight to download the patch, install it and test it
 
  • Like
Reactions: spirit

grimsrue

New Member
Jul 29, 2022
3
1
3
@grimsrue

can you test this patched version:

Code:
wget https://mutulin1.odiso.net/libpve-network-perl_0.7.1_all.deb
dpkg -i libpve-network-perl_0.7.1_all.deb

try with mixed ipv4/ipv6 peers , only ipv4 , only ipv6 if possible.

@spirit

I had a little time to download your patch and test it. It looks like it is working well.

One suggestion to the UI window for BGP is to add v4/v6 to the end or beginning of Peers so people know that they should be all IPv4 and IPv6 peers into the same field. Same for EVPN.

1661878075602.png


1661878365654.png
 

spirit

Famous Member
Apr 2, 2010
5,780
675
133
www.odiso.com
@spirit

I had a little time to download your patch and test it. It looks like it is working well.

One suggestion to the UI window for BGP is to add v4/v6 to the end or beginning of Peers so people know that they should be all IPv4 and IPv6 peers into the same field. Same for EVPN.

View attachment 40529


View attachment 40531
ok thanks for testing ! It'll send patch to pve-devel mailing list to include it in next release. I'look to improve the gui too.
 

lp_xanclas

New Member
Apr 29, 2022
10
1
3
Hi,
Are any docs mentioning how to setup BGP controller (with eBGP) to advertise the subnets on the VNETs to an external router?
I managed to receive external advertised subnets on the proxmox server, from a different AS that is configured on the proxmox BGP controller, but I don't know whow to advertise, on proxmox, the VNETs to that external AS.
Thank you.
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,780
675
133
www.odiso.com
Hi,
Are any docs mentioning how to setup BGP controller (with eBGP) to advertise the subnets on the VNETs to an external router?
I managed to receive external advertised subnets on the proxmox server, from a different AS that is configured on the proxmox BGP controller, but I don't know whow to advertise, on proxmox, the VNETs to that external AS.
Thank you.
I really need to write some docs with differents exemples.

you need:
1) enable exit-nodes on the zone options.
2) add an extra bgp controller for each exit-node, where on the peers options, you defines all hypervisors ip (like on the evpn controller) + the ip of your external routers + enabled the ebgp checkbox if your external as is different
 
  • Like
Reactions: lp_xanclas

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!