proxmox 7.0 sdn beta test
- Thread starter spirit
- Start date
Hello,
@spirit apologize to mark you here, but my setup seems quite strange to work with multiple proxmox nodes and SDN. The issue is, when I configure a BGP controller (with a different AS number from the AS in EVPN controller) I stop receiving the MAC address/VTEP association on the proxmox hosts. This means, communication across nodes in the same L2 doesn't work. I googled in and come in with this post:https://forum.proxmox.com/threads/proxmox-7-0-sdn-beta-test.69655/page-24
which made me configure two BGP controllers with different AS and checking the EBGP. On my vyos router I configured the BGP for my two proxmox hosts neighbors, and everything is working now. However I have this ip route table on my vyos router:
B>* 10.3.0.0/24 [20/0] via 10.2.1.3, eth2, weight 1, 00:00:11
B>* 10.3.0.10/32 [20/0] via 10.2.1.4, eth2, weight 1, 00:00:11
B>* 10.3.0.11/32 [20/0] via 10.2.1.3, eth2, weight 1, 00:00:11
which the /32 represents two containers in the different proxmox hosts. Should this be the expected behavior, that each single host address be advertised to my router? (my 10.3.0.0/24 is my VNET on proxmox SDN environment)
Probably you will need more details, but feel free to ask.
Thank you.
@spirit apologize to mark you here, but my setup seems quite strange to work with multiple proxmox nodes and SDN. The issue is, when I configure a BGP controller (with a different AS number from the AS in EVPN controller) I stop receiving the MAC address/VTEP association on the proxmox hosts. This means, communication across nodes in the same L2 doesn't work. I googled in and come in with this post:https://forum.proxmox.com/threads/proxmox-7-0-sdn-beta-test.69655/page-24
which made me configure two BGP controllers with different AS and checking the EBGP. On my vyos router I configured the BGP for my two proxmox hosts neighbors, and everything is working now. However I have this ip route table on my vyos router:
B>* 10.3.0.0/24 [20/0] via 10.2.1.3, eth2, weight 1, 00:00:11
B>* 10.3.0.10/32 [20/0] via 10.2.1.4, eth2, weight 1, 00:00:11
B>* 10.3.0.11/32 [20/0] via 10.2.1.3, eth2, weight 1, 00:00:11
which the /32 represents two containers in the different proxmox hosts. Should this be the expected behavior, that each single host address be advertised to my router? (my 10.3.0.0/24 is my VNET on proxmox SDN environment)
Probably you will need more details, but feel free to ask.
Thank you.
Last edited:
Hi, do you really need to use a differents AS for each node ?
using ebgp with different asn is a little bit more complex, until you have a lot of nodes or a l3 network underlay with bgp, I could recommand to keep same asn in the bgpcontroller
Not sure if it's a bug, but can you send the /etc/pve/sdn/controllers.cfg ?
(Note that, in the peers of bgp controller, you should still have all the proxmox nodes ips + your vyos ip)
yes, evpn announce /32 for each vm/ct.
(you can also enable the option "advertise subnet" on the zone, to send the full subnet route)
Hello @spirit after changing the both BGP controllers for each node to use the same AS used in vyos router, and added the IPs of the hosts in the both BGP controllers, I'm not able to reach some containers, and now even the VTEP/MAC exchanges are not being made. So I can't communicate from one container in one host to another container in another host, in the same VNET and address space.
Below are the controllers.cfg file on both servers:
1st host:
evpn: evpn01
asn 64512
peers 10.2.1.3,10.2.1.4
bgp: bgppve02
asn 65003
node pve02
peers 10.2.1.1,10.2.1.3,10.2.1.4
bgp-multipath-as-path-relax 0
ebgp 0
bgp: bgppve04
asn 65003
node pve04
peers 10.2.1.1,10.2.1.3,10.2.1.4
bgp-multipath-as-path-relax 0
ebgp 0
2nd host:
evpn: evpn01
asn 64512
peers 10.2.1.3,10.2.1.4
bgp: bgppve02
asn 65003
node pve02
peers 10.2.1.1,10.2.1.3,10.2.1.4
bgp-multipath-as-path-relax 0
ebgp 0
bgp: bgppve04
asn 65003
node pve04
peers 10.2.1.1,10.2.1.3,10.2.1.4
bgp-multipath-as-path-relax 0
ebgp 0
Probably many people request you, but do you have the full guide to configure this with EVPN and BGP controllers?
Thank you.
Below are the controllers.cfg file on both servers:
1st host:
evpn: evpn01
asn 64512
peers 10.2.1.3,10.2.1.4
bgp: bgppve02
asn 65003
node pve02
peers 10.2.1.1,10.2.1.3,10.2.1.4
bgp-multipath-as-path-relax 0
ebgp 0
bgp: bgppve04
asn 65003
node pve04
peers 10.2.1.1,10.2.1.3,10.2.1.4
bgp-multipath-as-path-relax 0
ebgp 0
2nd host:
evpn: evpn01
asn 64512
peers 10.2.1.3,10.2.1.4
bgp: bgppve02
asn 65003
node pve02
peers 10.2.1.1,10.2.1.3,10.2.1.4
bgp-multipath-as-path-relax 0
ebgp 0
bgp: bgppve04
asn 65003
node pve04
peers 10.2.1.1,10.2.1.3,10.2.1.4
bgp-multipath-as-path-relax 0
ebgp 0
But if you see, the subnet is already being advertised. With the "advertise subnet" checkbox unchecked.
Probably many people request you, but do you have the full guide to configure this with EVPN and BGP controllers?
Thank you.
Last edited:
are you using proxmox8 or proxmox7 ? because they are a bug currently in frr in version8 with some kind of routes missing. (I have a fixed frr version for testing)
Hello @spirit ,
I'm using proxmox 7. I'm not sure what the frr version I'm using, but once I get home I can check. But in the BGP and EVPN controllers I must use the same AS number?
I put it work by using the same AS number in everything (also in the vyos router). And using only one exit node.
I'm using proxmox 7. I'm not sure what the frr version I'm using, but once I get home I can check. But in the BGP and EVPN controllers I must use the same AS number?
I put it work by using the same AS number in everything (also in the vyos router). And using only one exit node.
Last edited:
ok, proxmox7/frr8.2.2, this version is fine.
I'll try to do a lab next week to reproduce your setup, (I'm a bit busy currently)
Hello @spirit , thanks for the help. I have three proxmox servers, I'm currently using a AS number for the EVPN controller and the a different AS number for the 3x BGP controllers that are connected to my vyos router. The vyos AS is the same of the BGP controllers on proxmox servers.
I'm using only one Exit node on my Zone. I can reach all the VMs hosted on exit node, but containers and other VMs on the others two proxmox servers I unable to reach. My vyos only gets the subnet advertisement at this time.
And on the nodes that are not working, I can't get the MAC/VTEP table for the VMs on the hosts.
This command in one the hosts shows nothing:
pXXXX# show evpn mac vni all
pXXXX#
I can't have connection between VMs/CT in the same segment on different nodes.
I'm using only one Exit node on my Zone. I can reach all the VMs hosted on exit node, but containers and other VMs on the others two proxmox servers I unable to reach. My vyos only gets the subnet advertisement at this time.
And on the nodes that are not working, I can't get the MAC/VTEP table for the VMs on the hosts.
This command in one the hosts shows nothing:
pXXXX# show evpn mac vni all
pXXXX#
I can't have connection between VMs/CT in the same segment on different nodes.
Hi spirit.
I'm back again seeking some guidance on how I can configure external exit nodes. Here's my configuration thus far.
vyos01-02
Then, I setup the zone "evpn001" with the following settings:
Finally, I setup the vnets as follows:
Some of my research shows that they may need to be configured as route-reflectors for the Proxmox nodes. Additionally, they should announce a default gateway in the EVPN zone. Traffic will ideally travel from inside a VNET to a VYOS router, and then it will be routed to my external edge firewalls that will handle external connectivity and NAT.
While I don't necessarily need to know the exact commands to run on the VYOS routers, I do need to know what they are supposed to be doing on a technical level. What kind of configuration did you do to your Arista switches to get them to act as exit nodes?
I appreciate any advice you can give.
Thanks!
I'm back again seeking some guidance on how I can configure external exit nodes. Here's my configuration thus far.
Hosts
pve01-05eth0 192.168.20.11-15
vyos01-02
eth0 192.168.20.21-22
SDN
I've setup an EVPN controller "evpn001" with the following settings:ASN #: 65001
Peers: 192.168.20.21 192.168.20.22
Then, I setup the zone "evpn001" with the following settings:
Controller: evpn001
VRF-VXLAN Tag: 1001000
MTU: 1450 (external MTU 1500)
Finally, I setup the vnets as follows:
vn001001:
Zone: evpn001
Tag: 1001001
vn001002:
Zone: evpn001
Tag: 1001002
Issues
They two VYOS nodes will act as exit nodes for the EVPN zone, but I'm stuck on how I need to configure these to properly talk with the Proxmox nodes. I do apologize as I am very new to BGP/EVPN.Some of my research shows that they may need to be configured as route-reflectors for the Proxmox nodes. Additionally, they should announce a default gateway in the EVPN zone. Traffic will ideally travel from inside a VNET to a VYOS router, and then it will be routed to my external edge firewalls that will handle external connectivity and NAT.
While I don't necessarily need to know the exact commands to run on the VYOS routers, I do need to know what they are supposed to be doing on a technical level. What kind of configuration did you do to your Arista switches to get them to act as exit nodes?
I appreciate any advice you can give.
Thanks!
oh, ok, vyos can do evpn. (I have just read the doc, it's simply a debian with frr, with a special cli to manage configuration).
so, indeed, you just need to configure an evpn controller (no need extra bgp controller), and add vyos in the evpn controller peers list.
use same asn for vyos and your proxmox nodes.
in evpn zone, don't configure exit-node. (as it'll be your vyos).
I don't known how is working the conf in vyos, but the idea is to announce the default 0.0.0.0 through evpn type-5 route.
also on vyos, you need to create a l3vxlan iface with same 1001000 vxlanid.
I'll try to do test next week with vyos.
Do I use this in conjunction with setting a gateway in the SDN vnet?
Will VMs will use the vnet gateway as their gateway and then that gateway will forward to my announced type-5 route destination, or do I need to give the vyos node an IP address on the VXLAN directly to act as a gateway?
My apologies for looking for so much help. I'm still trying to learn the technology and somehow it continues to evade grasp.
Yes, exactly. Vm gateway is proxmox vnet ip (anycast ip , same ip everywhere). Then the traffic to vyos forwarded through the type5.
(basically, vyos is the exit-node here , instead using a proxmox node as exit-node)
Hello everyone,
I have a question regarding multi-tenancy using this SDN. How can this be achieved? I understand that zones can help with this, allowing duplicated address spaces, but when the traffic needs to leave the proxmox, should we use a different EVPN controller for each tenant for example and with a different ASN?
Thank you
I have a question regarding multi-tenancy using this SDN. How can this be achieved? I understand that zones can help with this, allowing duplicated address spaces, but when the traffic needs to leave the proxmox, should we use a different EVPN controller for each tenant for example and with a different ASN?
Thank you
each zone is a different vrf, so traffic is not routed between a vm from a zone to a vm in another zone.
you only need 1 controller, asn is not related here.
(1 tenant = 1 zone = 1 vrf)
your exit node (if it's an external evpn router) need to a have a vrf with l3vni vxlan (the vxlanid defined on the zone) for each zone.
If you use a proxmox node as exit-node, it should work out of the box. (promox exit-node will route traffic from the evpn zone vrf(s) to real network, but not between differents vrf)
(Of course, you can't have same subnets in differents vrf, if you want to route them outside)
Hello, I'm trying setup a vyos router with proxmox and SDN, to implement the multi-tenancy concept. Besides adding my vyos router to the peers on SDN, this is my configuration on the vyos router for two tenants: customB and tenantC:
this seems not working because I announce twice (in each VRF) the default route. What happens with this, is that when I setup customB, I'm able to reach its bridge br5000 with address: 10.0.0.1.
However, once I configure the tenantC, I'm able to reach its bridge br5002 with address: 10.0.0.1, but the on customB I'm no longer able to reach the customB vRF bridge.
When I configure the tenantC, I see in on of the Proxmox servers that the default route moved to the VRF tenant C route table, and no longer is in the CustomB route table.
This might be expected, and I can probably making some wrong configuration. Any help I will appreciate.
Thank you.
tagging also @spirit and @forsytheda .
Thank oy for the help.
Code:
set interfaces ethernet eth2 address 10.2.2.2/24
set interfaces ethernet eth2 description 'to proxmox VTEPs'
set interfaces ethernet eth2 mtu 1600
set protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set protocols bgp address-family l2vpn-evpn advertise-all-vni
set protocols bgp system-as 65003
set protocols bgp neighbor 10.2.2.4 peer-group ibgp
set protocols bgp neighbor 10.2.2.6 peer-group ibgp
set protocols bgp neighbor 10.2.2.7 peer-group ibgp
set protocols bgp parameters log-neighbor-changes
set protocols bgp peer-group ibgp address-family l2vpn-evpn
set protocols bgp peer-group ibgp remote-as 65003
set protocols bgp peer-group ibgp update-source eth2
set interfaces vxlan vxlan5000 mtu 1550
set interfaces vxlan vxlan5000 parameters nolearning
set interfaces vxlan vxlan5000 port 4789
set interfaces vxlan vxlan5000 source-address 10.2.2.2
set interfaces vxlan vxlan5000 vni 5000
### CUSTOMB ###
set vrf name customB protocols bgp address-family ipv4-unicast redistribute connected
set vrf name customB protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name customB protocols bgp system-as 65003
set vrf name customB protocols bgp address-family ipv4-unicast network 0.0.0.0/0
set vrf name customB table 5000
set vrf name customB vni 5000
set interfaces bridge br5000 address 10.0.0.1/16
set interfaces bridge br5000 description customB
set interfaces bridge br5000 member interface vxlan5000
set interfaces bridge br5000 vrf customB
set interfaces vxlan vxlan5002 mtu 1550
set interfaces vxlan vxlan5002 parameters nolearning
set interfaces vxlan vxlan5002 port 4789
set interfaces vxlan vxlan5002 source-address 10.2.2.2
set interfaces vxlan vxlan5002 vni 5002
### TENANTC ###
set vrf name tenantC protocols bgp address-family ipv4-unicast redistribute connected
set vrf name tenantC protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name tenantC protocols bgp system-as 65003
set vrf name tenantC protocols bgp address-family ipv4-unicast network 0.0.0.0/0
set vrf name tenantC table 5002
set vrf name tenantC vni 5002
set interfaces bridge br5002 address 10.0.0.1/16
set interfaces bridge br5002 description tenantC
set interfaces bridge br5002 member interface vxlan5002
set interfaces bridge br5002 vrf tenantCthis seems not working because I announce twice (in each VRF) the default route. What happens with this, is that when I setup customB, I'm able to reach its bridge br5000 with address: 10.0.0.1.
However, once I configure the tenantC, I'm able to reach its bridge br5002 with address: 10.0.0.1, but the on customB I'm no longer able to reach the customB vRF bridge.
When I configure the tenantC, I see in on of the Proxmox servers that the default route moved to the VRF tenant C route table, and no longer is in the CustomB route table.
This might be expected, and I can probably making some wrong configuration. Any help I will appreciate.
Thank you.
tagging also @spirit and @forsytheda .
Thank oy for the help.
Hello, I'd like to edit this, mentioning that if both VMs are in the same proxmox server, I can reach both bridges on each zone. And the two default routes advertised in each VRF through proxmox are also visible in the routing table of each VRF on Proxmox server.
Thank you
Last edited:
Hello again,
from what I found, the configuration on vyos works fine, and I have an issue on the proxmox host that is used as exit node by another zone. Seems having zones configured with a proxmox exit node and other zones without setting an exit node (to allow the traffic being forwarded by vyos router), the VMs on this proxmox host don't work. (I have also a BGP controller using that proxmox host node).
Thank you.
Last edited:
What do you want to do exactly with your vyos router ? Because you can't route from outside to 1 vrf with 10.0.0.1 and another vrf 10.0.0.1. (they are no magic session tracking).
Multitenancy with same subnets can only live inside their own vrf (or zone in proxmox).
but even with differents subnet, I think your config is wrong. (you should try to enable the exit-node option on proxmox, and look the generated /etc/frr/frr.conf ,as vyos seem to use frr too, it can give your some inspiration).
I could be great to have your promox sdn config too (/etc/pve/sdn/*.cfg)
you should announce the default 0.0.0.0 in each vrf evpn network (and not the real netework), you need an vxlan (l3vni) interface for each vrf (same vxlanid than defined the proxmox zone). (5000 && 5002 from your config)
on proxmox, you can do #vtysh -c "sh ip bgp l2vpn evpn" , you should see 0.0.0.0/0 routes with your vyos ip as gateway.
This evpn default route is not imported in proxmox kernel routing table. (you don't see in #ip route)
From that, vm will send traffic ton his local anycast gateway on the vnet, and the vnet will forward to vyos because of the evpn 0.0.0.0 route.
After that, on Vyos, you can allow traffic going between different vrf (the defaut vrf (the real world - and the tenant vrf for example). You need to a vrf leak (importing/extporting routes between vrfs).
This is done with "import vrf <othervrfname>" in the source vrf router
But this can works of course, if you use differents subnets.