proxmox 7.0 sdn beta test

Hello,

This feature is very neat and quite useful in for Proxmox. Thank you for your effort in this!

That said, I am trying to have the Proxmox node communicate via BGP-EVPN over eBGP sessions to my switches, however the " bgp-evpn " controller option seems to only favour iBGP sessions. From the SDN plugin documentation, it looks like the " bgp-controller " option can be used for eBGP sessions, but it seems as if the SDN plugin is not putting that configuration into FRR, which still has the " deleted from the SDN plugin bgp-evpn " controller settings, trying to establish an iBGP session instead.

It was mentioned much earlier in this forum post that I can manipulate FRR normally via " vtysh " .. however at what point should I worry about FRR being overwritten by the SDN plugin?

Also, would it be beneficial to allow EVPN BGP sessions to use either " internal " or " external " BGP ASNs, as opposed to the numerical ASN itself? Here is what I am referring to from the FRR documentation: http://docs.frrouting.org/en/latest...EER-interface-remote-as-internal-external-ASN
 
Hello,

This feature is very neat and quite useful in for Proxmox. Thank you for your effort in this!

That said, I am trying to have the Proxmox node communicate via BGP-EVPN over eBGP sessions to my switches, however the " bgp-evpn " controller option seems to only favour iBGP sessions. From the SDN plugin documentation, it looks like the " bgp-controller " option can be used for eBGP sessions, but it seems as if the SDN plugin is not putting that configuration into FRR, which still has the " deleted from the SDN plugin bgp-evpn " controller settings, trying to establish an iBGP session instead.

It was mentioned much earlier in this forum post that I can manipulate FRR normally via " vtysh " .. however at what point should I worry about FRR being overwritten by the SDN plugin?

Also, would it be beneficial to allow EVPN BGP sessions to use either " internal " or " external " BGP ASNs, as opposed to the numerical ASN itself? Here is what I am referring to from the FRR documentation: http://docs.frrouting.org/en/latest...EER-interface-remote-as-internal-external-ASN
Hi, it's possible to do ebgp (I'm doing it at work).

what you need to do:
- create an evpn controller with an AS
- create a bgp controller for each node with a different AS

here an example:

Code:
evpn: evpnctl
        asn 65001
        peers 10.0.0.1,10.0.0.2

bgp: bgpproxmox1
        asn 65010
        node proxmox1
        peers 192.168.0.1,192.168.0.2
        ebgp 1
        ebgp-multihop 3
        loopback dummy0

bgp: bgpproxmox2
        asn 65011
        node proxmox2
        peers 192.168.0.1,192.168.0.2
        ebgp 1
        ebgp-multihop 3
        loopback dummy0
(you can do it through the gui)

and zones.cfg

Code:
evpn: customer
        controller evpnctl
        vrf-vxlan 10000
        disable-arp-nd-suppression 1
        advertise-subnets 1
        mac XX:XX:XX:XX:XX:XX:XX
        mtu 1500

In my setup, my proxmox nodes peers are my top of rack switches (they are another AS, ebgp, they don't support evpn).
and evpn controller peers is my arista routers (they do evpn, and are exit gw).
But you could use same peers if you want, if you can do evpn directly on your switches.
" ebgp 1" add the "remote-as external"


with this kind of config (both evpn && bgp controller defined),
each proxmox node use the ASN defined in bgp controller, and the asn from evpn controller is used as "route-target".
in frr.conf
Code:
 address-family l2vpn evpn
  route-target import 65001:10000
  route-target export 65001:10000
  advertise ipv4 unicast
  advertise ipv6 unicast
 exit-address-family
!
where 65001 is the asn defined in evpn controller, and 10000 is the vxlan-id / l3vni defined in the zone option.


this generate an frr.conf like

Code:
vrf vrf_customer
 vni 10000
exit-vrf
!
router bgp 65010
 bgp router-id 192.168.0.100
 no bgp default ipv4-unicast
 coalesce-time 1000
 bgp network import-check
 no bgp ebgp-requires-policy
 bgp disable-ebgp-connected-route-check
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 3
 neighbor  192.168.0.1 peer-group BGP
 neighbor  192.168.0.2 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as external
 neighbor VTEP bfd
 neighbor VTEP ebgp-multihop 10
 neighbor VTEP update-source dummy0
 neighbor 10.0.0.1 peer-group VTEP
 neighbor 10.0.0.2 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  advertise-all-vni
  autort as 65001
 exit-address-family
!
router bgp 65010 vrf vrf_customer
 no bgp ebgp-requires-policy
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  route-target import 65001:10000
  route-target export 65001:10000
  advertise ipv4 unicast
  advertise ipv6 unicast
 exit-address-family
!
 
It was mentioned much earlier in this forum post that I can manipulate FRR normally via " vtysh " .. however at what point should I worry about FRR being overwritten by the SDN plugin?

Note that, it's possible to create a "/etc/frr/frr.conf.local" manually with your frr config, it'll override the generate conf from proxmox.
(but any manual change done with vtysh / write mem , with be overwriten on next sdn push, so you need to do a "cp frr.conf frr.conf.local" after each change with vtysh).

This is really for special config.
if you have really classic/simple need, I would like to implement the needed feature in the sdn plugin. (So just ask me ;))

Also, would it be beneficial to allow EVPN BGP sessions to use either " internal " or " external " BGP ASNs, as opposed to the numerical ASN itself? Here is what I am referring to from the FRR documentation: http://docs.frrouting.org/en/latest...EER-interface-remote-as-internal-external-ASN
it's an frr trick, simply use "external", it'll work with any different asn than your asn. (when you can big setups, this simplify config).
 
Last edited:
Hi spirit,

I just rebooted my test Proxmox node, and there is some differences in configuration. Meaning, the controllers.cfg and frr.conf are different:

controllers.cfg:
bgp: bgp01eng12 asn 4200000120 node 01eng12 peers 172.20.24.9 ebgp 1

vtysh:

! router bgp 4200000120 bgp router-id 172.20.24.8 no bgp default ipv4-unicast coalesce-time 1000 neighbor VTEP peer-group neighbor VTEP remote-as 4200000120 neighbor VTEP bfd neighbor 172.20.24.9 peer-group VTEP ! address-family l2vpn evpn neighbor VTEP activate advertise-all-vni exit-address-family !

What would be the appropriate way to resolve this if this occurs?
 
Hi spirit,

I just rebooted my test Proxmox node, and there is some differences in configuration. Meaning, the controllers.cfg and frr.conf are different:

controllers.cfg:
bgp: bgp01eng12 asn 4200000120 node 01eng12 peers 172.20.24.9 ebgp 1

vtysh:

! router bgp 4200000120 bgp router-id 172.20.24.8 no bgp default ipv4-unicast coalesce-time 1000 neighbor VTEP peer-group neighbor VTEP remote-as 4200000120 neighbor VTEP bfd neighbor 172.20.24.9 peer-group VTEP ! address-family l2vpn evpn neighbor VTEP activate advertise-all-vni exit-address-family !

What would be the appropriate way to resolve this if this occurs?
I also tried to remove the ' router bgp ' config in FRR, but the SDN plugin is not repopulating the config that I put in on the web GUI.
 
". Meaning, the controllers.cfg and frr.conf are different:"
mmm, if I remember, bgpplugin can't be used standalone currently, it's really need to have a evpn controller too.


I also tried to remove the ' router bgp ' config in FRR, but the SDN plugin is not repopulating the config that I put in on the web GUI.
so what is your full config ? (/etc/pve/sdn/*.cfg ?)
 
so what is your full config ? (/etc/pve/sdn/*.cfg ?)
Code:
root@01eng12:~# ls -hal /etc/pve/sdn/*.cfg
-rw-r----- 1 root www-data 74 Jan 21 13:20 /etc/pve/sdn/controllers.cfg
-rw-r----- 1 root www-data  0 Jan 18 14:47 /etc/pve/sdn/zones.cfg
root@01eng12:~# cat /etc/pve/sdn/controllers.cfg
bgp: bgp01eng12
    asn 4200000120
    node 01eng12
    peers 172.20.24.9
    ebgp 1

root@01eng12:~# cat /etc/pve/sdn/zones.cfg
root@01eng12:~#
 
Code:
root@01eng12:~# ls -hal /etc/pve/sdn/*.cfg
-rw-r----- 1 root www-data 74 Jan 21 13:20 /etc/pve/sdn/controllers.cfg
-rw-r----- 1 root www-data  0 Jan 18 14:47 /etc/pve/sdn/zones.cfg
root@01eng12:~# cat /etc/pve/sdn/controllers.cfg
bgp: bgp01eng12
    asn 4200000120
    node 01eng12
    peers 172.20.24.9
    ebgp 1

root@01eng12:~# cat /etc/pve/sdn/zones.cfg
root@01eng12:~#
you don't have a zone ?
the frr config can't be generated without an an evpn zone + an evpn controller. (as I said, it's not standalone currently, it's just just for evpn, for define custom differents value for each node, like different asn, peers,....)
 
@M K

If you really want to use the sdn to only manage simple frr bgp , without any zones/vnets/evpn, I can look to implement it. It shouldn't be too difficult.
Hey Spirit,

I guess what I am doing is trying to understand how the SDN plugin works.
I was under the impression that the SDN plugin would manipulate FRR, and it does to some extent. What I am afraid of the SDN plugin overwriting my FRR config, however you did mention to leverage the frr.config.local configuration file as a " hammer " to avoid the SDN plugin overwriting FRR. I just need time to try that out.

I will keep you posted.
I appreciate your feedback!
 
Just a progress update: I was able to get VXLAN working with eBGP EVPN peer and have a VM and CT communicate over that VXLAN tunnel on the EVPN network, so thats good. I am running into a minor issue with MTU (iperf3 will stop after the first second, however if I lower the TCP MSS on iperf3 it does work .. so I need to find out which interface to increase the MTU to jumbo frame).

Also, I had to force FRR to advertise the VTEP IP via establishing a BGP peer via IPv4 Unicast AFI in parallel with the L2VPN EVPN AFI.

Code:
root@01eng12:~# cat /etc/pve/sdn/*.cfg
evpn: evpn0112
    asn 4200000112
    peers 172.20.24.9

bgp: bgp01eng12
    asn 4200000112
    node 01eng12
    peers 172.20.24.9
    ebgp 1

vnet: vlan24
    zone vni24
    tag 24

evpn: vni24
    controller evpn0112
    vrf-vxlan 117
    exitnodes 01eng12
    ipam pve
    mac 8A:C4:CB:C6:A4:7B

root@01eng12:~#

root@01eng12:~# vtysh

Hello, this is FRRouting (version 8.0.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

01eng12# sh bgp sum

IPv4 Unicast Summary:
BGP router identifier 172.20.24.8, local AS number 4200000112 vrf-id 0
BGP table version 6
RIB entries 11, using 2024 bytes of memory
Peers 1, using 723 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor                       V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
01sw17(172.20.24.9) 4 4200000117      3815      3816        0    0    0 02:10:32            5        6 N/A

Total number of neighbors 1

L2VPN EVPN Summary:
BGP router identifier 172.20.24.8, local AS number 4200000112 vrf-id 0
BGP table version 0
RIB entries 7, using 1288 bytes of memory
Peers 1, using 723 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor                       V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
01sw17(172.20.24.9) 4 4200000117      3815      3816        0    0    0 02:10:32            8        9 N/A

Total number of neighbors 1
01eng12#
 
I am running into a minor issue with MTU (iperf3 will stop after the first second, however if I lower the TCP MSS on iperf3 it does work .. so I need to find out which interface to increase the MTU to jumbo frame).
yes, vxlan encapsulation need 50 more bytes.
So:
1) if your proxmox node physical interfaces are mtu=1500, you need to lower mtu to 1450 inside your guest vm. (proxmox currently don't auto change that for you, but I have some future patch to do it).

2) if you network support jumbo frame, you can increase mtu on your physical interface to 1550 or more. (on the nodes physical interfaces (ethX,bondX,..) where the vtep ip is located)


Also, I had to force FRR to advertise the VTEP IP via establishing a BGP peer via IPv4 Unicast AFI in parallel with the L2VPN EVPN AFI.
did you need to do any manual change in frr.conf ? sdn plugin create by default a 1 bgp session + 1 evpn session to the peers define in the evpn controller. (and it's possible to override bgp session peer,asn,... by creating a specific bgp controller for each node).
 
Love the SDN plugin, we're using it with great success! Fantastic work! Was previously using this method, but the plugin makes things even easier.

I did run across one concern, we cannot add a BGP controller with nodes that have a dash (-) in their hostname (i.e. myhostname-24):

Screen Shot 2022-01-28 at 4.26.54 PM.png

I know there is a couple of ways I can develop a work-around, but without altering your code further (I had to edit EvpnPlugin.pm on my exit node(s) as it inserting push @iface_config, "post-up ip route add vrf $vrf_iface unreachable default metric 4278198272";), do you have a solution?

Thanks again sooooo much for your great work on this!!! Keep it up!!!!!!!
 
Last edited:
I did run across one concern, we cannot add a BGP controller with nodes that have a dash (-) in their hostname (i.e. myhostname-24):
do you have a solution?
The fix would be easy, as it's just expanding the regex for the pve-sdn-controller-id format, and allowing minus and underscore there in the middle of the string shouldn't have any ramifications, FWICT, or do I overlooked something @spirit ?
 
The fix would be easy, as it's just expanding the regex for the pve-sdn-controller-id format, and allowing minus and underscore there in the middle of the string shouldn't have any ramifications, FWICT, or do I overlooked something @spirit ?
Thanks for your reply Thomas! I was thinking the same thing, as I did some research, and noticed PVE/Network/SDN/Controllers/Plugin.pm was like that previously.

I went ahead and modified it - and was able to create a BGP controller successfully:

Perl:
Changed PVE/Network/SDN/Controllers/Plugin.pm

from:

if ($id !~ m/^[a-z][a-z0-9]*[a-z0-9]$/i) {

to:

if ($id !~ m/^[a-z][a-z0-9\-\_\.]*[a-z0-9]$/i) {

However, when it went to reload frr, it gave me an error:

TASK ERROR: command '/usr/lib/frr/frr-reload.py --stdout --reload /etc/frr/frr.conf' failed: exit code 1

Perl:
root@myhostname-24:~# cat /etc/pve/sdn/*.cfg
evpn: evpn400
        asn 65001
        peers 192.168.1.18

bgp: bgpmyhostname-24
        asn 65001
        node myhostname-24
        peers 192.168.1.18
        ebgp 1

subnet: evpn400-10.10.10.160-27
        vnet pubnet2
        gateway 10.10.10.190

vnet: pubnet2
        zone evpn400
        tag 14000

evpn: evpn400
        controller evpn400
        vrf-vxlan 10001
        exitnodes myhostname-24
        ipam pve
        mac 9A:E1:F8:7C:A2:70
        mtu 1450

Where 192.168.1.18 is a route reflector running FRR.

Code:
myhostname-24# sh bgp sum

L2VPN EVPN Summary:
BGP router identifier 192.168.1.24, local AS number 65001 vrf-id 0
BGP table version 0
RIB entries 8, using 1536 bytes of memory
Peers 1, using 21 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor             V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
route-reflector(192.168.1.18) 4      65001      1479      1479        0    0    0 01:13:31            1        6

Route reflector (haven't created BGP controller for my other 192.168.1.25 node yet):

Code:
route-reflector# sh bgp sum

L2VPN EVPN Summary:
BGP router identifier 192.168.1.18, local AS number 65001 vrf-id 0
BGP table version 0
RIB entries 5, using 960 bytes of memory
Peers 2, using 43 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor               V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
*myhostname-24(192.168.1.24) 4      65001      1504      1504        0    0    0 01:14:43            6        7
*myhostname-25(192.168.1.25) 4      65001      1656      1670        0    0    0 01:22:38            1        7

Total number of neighbors 2
* - dynamic neighbor
2 dynamic neighbor(s), limit 100

I'm also wanting my proxmox nodes to announce the evpn Iv4 routes to the route reflector so I can peer it with my gateway/upstream routers.

Let me know how to handle, I greatly appreciate it!
 
The fix would be easy, as it's just expanding the regex for the pve-sdn-controller-id format, and allowing minus and underscore there in the middle of the string shouldn't have any ramifications, FWICT, or do I overlooked something @spirit ?
yes, only the regex need to be fixed. (don't have too much time this week, but I can send a patch next week)
 
However, when it went to reload frr, it gave me an error:

TASK ERROR: command '/usr/lib/frr/frr-reload.py --stdout --reload /etc/frr/frr.conf' failed: exit code 1

can you send your generated /etc/frr/frr.conf ?

do you have tried a full restart of frr ? (systemctl restart frr)
 
I'm also wanting my proxmox nodes to announce the evpn Iv4 routes to the route reflector so I can peer it with my gateway/upstream routers.

The vms ipv4 /32 or ipv6 /128 should be announced as evpn type3 routes.
It's possible to advertise the subnets defined in subnets.cfg with evpn type5 routes. It's not available in gui yet (I had send patchs some months ago, but it's not applied, I need to check that). But it should be possible to add the option in the zones.cfg.

evpn: evpn400
controller evpn400
vrf-vxlan 10001
exitnodes myhostname-24
ipam pve
advertise-subnets 1
mac 9A:E1:F8:7C:A2:70
mtu 1450


In your route reflector, you should be able to redistribute evpn routes into your classic bgp routes
 
can you send your generated /etc/frr/frr.conf ?

do you have tried a full restart of frr ? (systemctl restart frr)
Sure:

Code:
root@myhostname-24:~# cat /etc/frr/frr.conf
log syslog informational
ip forwarding
ipv6 forwarding
frr defaults datacenter
service integrated-vtysh-config
hostname myhostname-24
!
!
vrf vrf_evpn400
 vni 10001
exit-vrf
!
router bgp 65001
 bgp router-id 192.168.1.24
 no bgp default ipv4-unicast
 coalesce-time 1000
 bgp network import-check
 no bgp ebgp-requires-policy
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor 192.168.1.18 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65001
 neighbor VTEP bfd
 neighbor 192.168.1.18 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn400
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_evpn400
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
!
router bgp 65001 vrf vrf_evpn400
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
line vty

Code:
myhostname-24# show running-config
Building configuration...

Current configuration:
!
frr version 7.4
frr defaults datacenter
hostname myhostname-24
log syslog informational
service integrated-vtysh-config
!
vrf vrf_evpn400
 vni 10001
 exit-vrf
!
router bgp 65001
 bgp router-id 192.168.1.24
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65001
 neighbor VTEP bfd
 neighbor 192.168.1.18 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn400
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_evpn400
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
!
router bgp 65001 vrf vrf_evpn400
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
line vty
!
end

Just did a restart and it looks like I need to modify my Route reflector config:

Code:
Jan 31 11:25:21 myhostname-24 bgpd[20937]: [EC 33554465] 192.168.1.18 [FSM] unexpected packet received in state OpenSent
Jan 31 11:25:21 myhostname-24 bgpd[20937]: %NOTIFICATION: sent to neighbor 192.168.1.18 5/1 (Neighbor Events Error/Receive Unexpected Message in Op
Jan 31 11:25:22 myhostname-24 bgpd[20937]: %NOTIFICATION: received from neighbor 192.168.1.18 2/7 (OPEN Message Error/Unsupported Capability) 6 byt
Jan 31 11:25:22 myhostname-24 bgpd[20937]: [EC 33554465] 192.168.1.18 [FSM] unexpected packet received in state OpenSent
Jan 31 11:25:22 myhostname-24 bgpd[20937]: %NOTIFICATION: sent to neighbor 192.168.1.18 5/1 (Neighbor Events Error/Receive Unexpected Message in Op
Jan 31 11:25:23 myhostname-24 bgpd[20937]: %NOTIFICATION: received from neighbor 192.168.1.18 2/7 (OPEN Message Error/Unsupported Capability) 6 byt
Jan 31 11:25:23 myhostname-24 bgpd[20937]: [EC 33554465] 192.168.1.18 [FSM] unexpected packet received in state OpenSent
Jan 31 11:25:23 myhostname-24 bgpd[20937]: %NOTIFICATION: sent to neighbor 192.168.1.18 5/1 (Neighbor Events Error/Receive Unexpected Message in Op
Jan 31 11:25:24 myhostname-24 bgpd[20937]: %NOTIFICATION: sent to neighbor 192.168.1.18 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes f
Jan 31 11:25:24 myhostname-24 bgpd[20937]: [EC 33554451] bgp_process_packet: BGP OPEN receipt failed for peer: 192.168.1.18
lines 1-25/25 (END)

Code:
route-reflector# show running-config
Building configuration...

Current configuration:
!
frr version 7.5
frr defaults datacenter
hostname route-reflector
log file /var/log/frr/frr.log
log syslog informational
service integrated-vtysh-config
!
router bgp 65001
 bgp router-id 192.168.1.18
 no bgp default ipv4-unicast
 bgp cluster-id 192.168.1.18
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65001
 neighbor VTEP update-source 192.168.1.18
 neighbor VTEP capability extended-nexthop
 bgp listen range 192.168.1.0/24 peer-group VTEP
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-reflector-client
 exit-address-family
!
line vty
!
end
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!