Implementations of SDN Networking

U

u432897

Member
Dec 15, 2020
7
0
6
34
Hi all,

I'm curious if those in the forums have had luck virtualizing networks with the SDN feature, something along the lines of using a firewall/router to route traffic completely within a node. The use case would be to remove physical interface dependencies if traffic doesn't need to leave the host (e.g., virtual firewall on host routes between subnets) and make backup/restores of a complete host (to a new host) easier.

Additionally, is there any information when the SDN feature will be outside of the technical preview and/or supported?

Thanks!
 
u432897 said:
Hi all,

I'm curious if those in the forums have had luck virtualizing networks with the SDN feature, something along the lines of using a firewall/router to route traffic completely within a node. The use case would be to remove physical interface dependencies if traffic doesn't need to leave the host (e.g., virtual firewall on host routes between subnets) and make backup/restores of a complete host (to a new host) easier.
Click to expand...
Hi,
as I am the main dev of the sdn feature, I can say, yes, I'm using in production. (I'm working for an hosting company with our own infrastructure).

So , we are using bgp-evpn sdn with anycast gateway (private && public network), promxox firewall, ceph, on multiple datacenters. (so we can easily move a vm from a DC to another DC)

On only last physical network hardware is our main arista bgp routers to manage internet transit. (and they are able to do bgp-evpn too, so they are the exit nodes of our evpn network too).
 
Thank you for the reply and supporting the software!

I'll work on implementing the use case I described above knowing it's used in a much more production environment than my home lab... I hope to use OPNSense to route between networks and enforce firewall policy without complex physical networking that makes restores challenging.
 
If you use opnsense, it should it be easy with any sdn implementation (vlan, vxlan,...).

I have some plan (in the future, no roadmap yet), to auto manage sdn gateway like pfsense, or other. (pushing subnets/gw ip to the appliance).
 
spirit said:
Hi,
as I am the main dev of the sdn feature, I can say, yes, I'm using in production. (I'm working for an hosting company with our own infrastructure).

So , we are using bgp-evpn sdn with anycast gateway (private && public network), promxox firewall, ceph, on multiple datacenters. (so we can easily move a vm from a DC to another DC)

On only last physical network hardware is our main arista bgp routers to manage internet transit. (and they are able to do bgp-evpn too, so they are the exit nodes of our evpn network too).
Click to expand...
Hi, old thread but what's what I trying to achieve.

Would you be so kind to elaborate a bit how you did this and what did you guys used for anycast gateway.

In my lab I getting the bgp working okej. It pushes and getting the routes from the firewall/router.
I am able to ping the WM.
But when I add multiple exit nodes the traffic stops. But the routes for the WM is propagated.
 
ic3cool said:
Hi, old thread but what's what I trying to achieve.

Would you be so kind to elaborate a bit how you did this and what did you guys used for anycast gateway.

In my lab I getting the bgp working okej. It pushes and getting the routes from the firewall/router.
I am able to ping the WM.
But when I add multiple exit nodes the traffic stops. But the routes for the WM is propagated.
Click to expand...
Hi,
so it's working fine when you only have only 1 exit-node ?

The anycast gateway is the ip defined on the vnet itself (same ip on same vnet on differents proxmox nodes).

so the traffic is to go outside is:

vm------->anycast gateway(vnet)---------(announced default 0.0.0.0/0)------->exit-node(s)-------(bgp or static route)-----------------> router

and the reverse way

router------(arrounced route or static route to your evpn subnets)----------->exit-node(s)-------------->anycastgatway(vnet)----------> vm
 
spirit said:
Hi,
so it's working fine when you only have only 1 exit-node ?

The anycast gateway is the ip defined on the vnet itself (same ip on same vnet on differents proxmox nodes).

so the traffic is to go outside is:

vm------->anycast gateway(vnet)---------(announced default 0.0.0.0/0)------->exit-node(s)-------(bgp or static route)-----------------> router

and the reverse way

router------(arrounced route or static route to your evpn subnets)----------->exit-node(s)-------------->anycastgatway(vnet)----------> vm
Click to expand...
Hi,
Yes and it does not matter which of the tree nodes, its working one by one.
bgp is working communication wise but its looks like its not preferring the the host where the VM is located.

How did you set up your evpn? did you dedicated a bridge on each host and put the ip directly to that or are you using an vlan interface?
 
ic3cool said:
Hi,
Yes and it does not matter which of the tree nodes, its working one by one.
bgp is working communication wise but its looks like its not preferring the the host where the VM is located.
Click to expand...
indeed, if you use bgp between your router and exit nodes, any exit nodes will be used. (and then exit-node forward traffic through evpn to to final destination node where the vm is located).

They are no way to change that, until your router support evpn directly (so your router is the exit-node directly)

(Personnally, at work, I'm use arista switches as exit gateway)

ic3cool said:
How did you set up your evpn? did you dedicated a bridge on each host and put the ip directly to that or are you using an vlan interface?
Click to expand...
I have both method, some of my node have ip directly on an vlan interface, some others on vmbrX.
It's not important, as vxlan encapsulation is done over tcp/ip - layer3.
 
spirit said:
indeed, if you use bgp between your router and exit nodes, any exit nodes will be used. (and then exit-node forward traffic through evpn to to final destination node where the vm is located).

They are no way to change that, until your router support evpn directly (so your router is the exit-node directly)

(Personnally, at work, I'm use arista switches as exit gateway)


I have both method, some of my node have ip directly on an vlan interface, some others on vmbrX.
It's not important, as vxlan encapsulation is done over tcp/ip - layer3.
Click to expand...

The odd thing happens when I add more then one exit-node is introduced.

I've done some tests with the routing and it's working fine.

Its almost like it stops forward traffic when more then one node.

Is there any good debug mode or somewhere to look for hints what might happen to the setup when multiple nodes are up?
 
By the way i did another "small" test around the Exit-thing.

Two BR and One VLAN setup.

VLAN raw to BR1 with physical nic connected and VLAN member to BR2

Set the IP for each host on BR2. Connection is up.

One exit node is working. VMs are able to ping from one to another on different host.. fine

Here comes the test :)

Remove VLAN from BR2. Expected hosts can't be reached in between on that subnet.
But VM1 and VM2 is able to ping another gateway on a different lan from different hosts with multiple exit-node enabled and ofc not eachother via vxlan
 
ic3cool said:
Is there any good debug mode or somewhere to look for hints what might happen to the setup when multiple nodes are up?
Click to expand...
when multiple exit-node are defined, they both announce 0.0.0.0/0 in the evpn network. That's the only difference.
So, it should do ecmp inside the evpn network (both exit-node have same weight for 0.0.0.0/0 route).

Maybe the problem is it between the router && the exit-nodes ?


y the way i did another "small" test around the Exit-thing.

Two BR and One VLAN setup.

VLAN raw to BR1 with physical nic connected and VLAN member to BR2

Set the IP for each host on BR2. Connection is up.

One exit node is working. VMs are able to ping from one to another on different host.. fine
Click to expand...

can you send you /etc/pve/sdn/*.cfg && /etc/network/interfaces && /etc/network/interfaces.d/sdn of differents nodes ?

and also the result of:
vtysh -c "sh bgp l2vpn evpn"

on 1 of the node.
 
Last edited:
Hi sorry for my delay.

for now i have removed BGP connection to ease up the configuration and complexity a bit. just to get this exit node thing sorted.

the nodes is also up2date with the latest today.

when multiple exit-node are defined, they both announce 0.0.0.0/0 in the evpn network. That's the only difference.
So, it should do ecmp inside the evpn network (both exit-node have same weight for 0.0.0.0/0 route).

Maybe the problem is it between the router && the exit-nodes ?
Click to expand...

i've tested them one by one and its works.
also running plenty of vms "legacy"-Vlan style and working.

and also the result of:
vtysh -c "sh bgp l2vpn evpn"

on 1 of the node.
Click to expand...

vtysh -c "sh bgp l2vpn evpn"
BGP table version is 14, local router ID is 192.168.89.11
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.100.1:2
* i[5]:[0]:[0]:[0.0.0.0]
192.168.89.10(parker)
100 0 i
RT:65000:20000 ET:8 Rmac:4e:80:72:11:5e:12
*> 192.168.89.11(stark)
32768 i
ET:8 RT:65000:20000 Rmac:52:b7:46:eb:3a:c4
* i 192.168.89.21(bruce)
100 0 i
RT:65000:20000 ET:8 Rmac:52:1c:f4:59:e9:43
* i[5]:[0]:[0]:[::] 192.168.89.10(parker)
100 0 i
RT:65000:20000 ET:8 Rmac:4e:80:72:11:5e:12
*> 192.168.89.11(stark)
32768 i
ET:8 RT:65000:20000 Rmac:52:b7:46:eb:3a:c4
* i 192.168.89.21(bruce)
100 0 i
RT:65000:20000 ET:8 Rmac:52:1c:f4:59:e9:43
Route Distinguisher: 192.168.89.10:3
*>i[2]:[0]:[48]:[ba:54:48:61:98:df]
192.168.89.10(parker)
100 0 i
RT:65000:10000 ET:8 MM:2
*>i[3]:[0]:[32]:[192.168.89.10]
192.168.89.10(parker)
100 0 i
RT:65000:10000 ET:8
Route Distinguisher: 192.168.89.10:4
*>i[2]:[0]:[48]:[de:5e:89:90:fb:87]
192.168.89.10(parker)
100 0 i
RT:65000:10001 ET:8 MM:1
*>i[2]:[0]:[48]:[de:5e:89:90:fb:87]:[32]:[10.0.100.12]
192.168.89.10(parker)
100 0 i
RT:65000:10001 RT:65000:20000 ET:8 MM:1 Rmac:4e:80:72:11:5e:12
*>i[2]:[0]:[48]:[de:5e:89:90:fb:87]:[128]:[fe80::dc5e:89ff:fe90:fb87]
192.168.89.10(parker)
100 0 i
RT:65000:10001 ET:8 MM:1
*>i[3]:[0]:[32]:[192.168.89.10]
192.168.89.10(parker)
100 0 i
RT:65000:10001 ET:8
Route Distinguisher: 192.168.89.11:3
*> [2]:[0]:[48]:[de:c0:16:71:b9:aa]
192.168.89.11(stark)
32768 i
ET:8 RT:65000:10000 MM:2
*> [3]:[0]:[32]:[192.168.89.11]
192.168.89.11(stark)
32768 i
ET:8 RT:65000:10000
Route Distinguisher: 192.168.89.11:4
*> [2]:[0]:[48]:[ae:13:ee:64:b0:b1]
192.168.89.11(stark)
32768 i
ET:8 RT:65000:10001
*> [2]:[0]:[48]:[ae:13:ee:64:b0:b1]:[32]:[10.0.100.11]
192.168.89.11(stark)
32768 i
ET:8 RT:65000:10001 RT:65000:20000 Rmac:52:b7:46:eb:3a:c4
*> [2]:[0]:[48]:[ae:13:ee:64:b0:b1]:[128]:[fe80::ac13:eeff:fe64:b0b1]
192.168.89.11(stark)
32768 i
ET:8 RT:65000:10001
*> [3]:[0]:[32]:[192.168.89.11]
192.168.89.11(stark)
32768 i
ET:8 RT:65000:10001
Route Distinguisher: 192.168.89.21:3
*>i[3]:[0]:[32]:[192.168.89.21]
192.168.89.21(bruce)
100 0 i
RT:65000:10000 ET:8
Route Distinguisher: 192.168.89.21:4
*>i[2]:[0]:[48]:[6a:c9:3a:e2:9d:e2]
192.168.89.21(bruce)
100 0 i
RT:65000:10001 ET:8
*>i[2]:[0]:[48]:[6a:c9:3a:e2:9d:e2]:[32]:[10.0.100.10]
192.168.89.21(bruce)
100 0 i
RT:65000:10001 RT:65000:20000 ET:8 Rmac:52:1c:f4:59:e9:43
*>i[2]:[0]:[48]:[6a:c9:3a:e2:9d:e2]:[128]:[fe80::68c9:3aff:fee2:9de2]
192.168.89.21(bruce)
100 0 i
RT:65000:10001 ET:8
*>i[3]:[0]:[32]:[192.168.89.21]
192.168.89.21(bruce)
100 0 i
RT:65000:10001 ET:8

Displayed 19 out of 23 total prefixes
 

Attachments

BGP table version is 18, local router ID is 192.168.89.11 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP] EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 10.0.100.1:2 * i[5]:[0]:[0]:[0.0.0.0] 192.168.89.10(parker) 100 0 i RT:65000:20000 ET:8 Rmac:4e:80:72:11:5e:12 *> 192.168.89.11(stark) 32768 i ET:8 RT:65000:20000 Rmac:52:b7:46:eb:3a:c4 * i 192.168.89.21(bruce) 100 0 i RT:65000:20000 ET:8 Rmac:52:1c:f4:59:e9:43 * i[5]:[0]:[0]:[::] 192.168.89.10(parker) 100 0 i RT:65000:20000 ET:8 Rmac:4e:80:72:11:5e:12 *> 192.168.89.11(stark) 32768 i ET:8 RT:65000:20000 Rmac:52:b7:46:eb:3a:c4 * i 192.168.89.21(bruce) 100 0 i RT:65000:20000 ET:8 Rmac:52:1c:f4:59:e9:43 Route Distinguisher: 192.168.89.10:3 *>i[2]:[0]:[48]:[ba:54:48:61:98:df] 192.168.89.10(parker) 100 0 i RT:65000:10000 ET:8 MM:2 *>i[3]:[0]:[32]:[192.168.89.10] 192.168.89.10(parker) 100 0 i RT:65000:10000 ET:8 Route Distinguisher: 192.168.89.10:4 *>i[2]:[0]:[48]:[de:5e:89:90:fb:87] 192.168.89.10(parker) 100 0 i RT:65000:10001 ET:8 *>i[2]:[0]:[48]:[de:5e:89:90:fb:87]:[32]:[10.0.100.12] 192.168.89.10(parker) 100 0 i RT:65000:10001 RT:65000:20000 ET:8 Rmac:4e:80:72:11:5e:12 *>i[2]:[0]:[48]:[de:5e:89:90:fb:87]:[128]:[fe80::dc5e:89ff:fe90:fb87] 192.168.89.10(parker) 100 0 i RT:65000:10001 ET:8 *>i[3]:[0]:[32]:[192.168.89.10] 192.168.89.10(parker) 100 0 i RT:65000:10001 ET:8 Route Distinguisher: 192.168.89.11:3 *> [2]:[0]:[48]:[de:c0:16:71:b9:aa] 192.168.89.11(stark) 32768 i ET:8 RT:65000:10000 MM:2 *> [3]:[0]:[32]:[192.168.89.11] 192.168.89.11(stark) 32768 i ET:8 RT:65000:10000 Route Distinguisher: 192.168.89.11:4 *> [2]:[0]:[48]:[ae:13:ee:64:b0:b1] 192.168.89.11(stark) 32768 i ET:8 RT:65000:10001 *> [2]:[0]:[48]:[ae:13:ee:64:b0:b1]:[32]:[10.0.100.11] 192.168.89.11(stark) 32768 i ET:8 RT:65000:10001 RT:65000:20000 Rmac:52:b7:46:eb:3a:c4 *> [2]:[0]:[48]:[ae:13:ee:64:b0:b1]:[128]:[fe80::ac13:eeff:fe64:b0b1] 192.168.89.11(stark) 32768 i ET:8 RT:65000:10001 *> [3]:[0]:[32]:[192.168.89.11] 192.168.89.11(stark) 32768 i ET:8 RT:65000:10001 Route Distinguisher: 192.168.89.21:3 *>i[3]:[0]:[32]:[192.168.89.21] 192.168.89.21(bruce) 100 0 i RT:65000:10000 ET:8 Route Distinguisher: 192.168.89.21:4 *>i[3]:[0]:[32]:[192.168.89.21] 192.168.89.21(bruce) 100 0 i RT:65000:10001 ET:8 Displayed 16 out of 20 total prefixes
 
ok, I'm not sure, but I see that you use the snat feature.

I think it can't work correctly with multiple exit-node, as each node have same weight, so traffic is balanced between node,
but tracking of nat can't work correctly.

one solution could be to implement conntrackd to sync nodes. (but maybe it could conflict with proxmox firewall)

another solution, could be to have a bigger weight on 1 exit-node (I need to code that in sdn plugin), to nat always occur on 1 node, and failover.
and on the reverse direction, maybe evpn subnets could be announce from 1exit-node only.

I'll try to do tests my side next week.
 
hi,
thank you for putting time in this.

ive enabled BGP and as before all three nodes announces and connects to the external router fine.
i also tried to disable SNAT.

same result as before multiple exits not working, specifying only one and its working from inside and also from outside.
 
ic3cool said:
hi,
thank you for putting time in this.

ive enabled BGP and as before all three nodes announces and connects to the external router fine.
i also tried to disable SNAT.

same result as before multiple exits not working, specifying only one and its working from inside and also from outside.
Click to expand...
ok. without snat, I really don't see what could be the problem, but I need to do more test to be sure. I'll keep you in touch next week.
 
I seem to be getting the same issue as above. I have bgp-evpn setup working and this distributes my subnets to my switch via plain ipv4 BGP (made this by adding the BGP controller on the exit node). With one exit node, everything works apart from the fact that the outside cannot ping any nodes on a host that isn't the exit node.

To fix that last issue, on my node that wasn't an exit node, I copied the BGP config for the vrf over from the other node and took out the "default originate", although this isn't really a fix as any change I make to the SDN will overwrite this change. I wish there was an option to make a node that does this.

With the exit node, I really don't know how to fix it. I enable it, with or without source NAT, and suddenly the VMs can only ping other VMs on the same network. Maybe i need the proxmox hosts themselves to be on different, routed ports on my switch for it to work, but with them on the same subnet it just breaks everything.
 
lunam said:
I seem to be getting the same issue as above. I have bgp-evpn setup working and this distributes my subnets to my switch via plain ipv4 BGP (made this by adding the BGP controller on the exit node). With one exit node, everything works apart from the fact that the outside cannot ping any nodes on a host that isn't the exit node.


With the exit node, I really don't know how to fix it. I enable it, with or without source NAT, and suddenly the VMs can only ping other VMs on the same network. Maybe i need the proxmox hosts themselves to be on different, routed ports on my switch for it to work, but with them on the same subnet it just breaks everything.
Click to expand...
in last sdn version, I have have weight priority option when multiple exit-nodes are used. it's not yet available in gui, but with libpvenetwork 0.7,

you can already edit : /etc/pve/sdn/zones.cfg, and add " exitnodes-primary"

Code: 
evpn: myzon
    controller evpnctl
    vrf-vxlan 10000
    exitnodes node1,node2
    exitnodes-primary node1
    mtu 1500


The problem with Snat, is that we use iptables conntrack to track the current established connection, so packets can't be loadbalanced between 2 exit-nodes.
with exitnodes-primary, i's active-backup. so packets always going out through node1, and if node1 is down, it's going to node2.



To fix that last issue, on my node that wasn't an exit node, I copied the BGP config for the vrf over from the other node and took out the "default originate", although this isn't really a fix as any change I make to the SDN will overwrite this change. I wish there was an option to make a node that does this.
Click to expand...
I don't understand. "default originate" is only added on exit-node. So simply don't define this node as "exit-node" in the zone configuration.
 
spirit said:
in last sdn version, I have have weight priority option when multiple exit-nodes are used. it's not yet available in gui, but with libpvenetwork 0.7,

you can already edit : /etc/pve/sdn/zones.cfg, and add " exitnodes-primary"

Code: 
evpn: myzon
    controller evpnctl
    vrf-vxlan 10000
    exitnodes node1,node2
    exitnodes-primary node1
    mtu 1500


The problem with Snat, is that we use iptables conntrack to track the current established connection, so packets can't be loadbalanced between 2 exit-nodes.
with exitnodes-primary, i's active-backup. so packets always going out through node1, and if node1 is down, it's going to node2.




I don't understand. "default originate" is only added on exit-node. So simply don't define this node as "exit-node" in the zone configuration.
Click to expand...
I tried adding the exitnodes-primary, and it didn't make a difference. Exact same issues where the nodes can't reach anything but themselves.

What I mean was I added this to the non-exit nodes to make it work, sorry I wasn't very clear in my original comment:

Code: 
router bgp 65000 vrf vrf_main
 bgp router-id 10.0.0.210
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family

When I tried using the local routing setting, suddenly no routes were advertised to my switch via BGP anymore. The advertise subnets also didn't seems to make a difference either.

To fix it now (at least for my use case), I have advertise subnets checked and then on my 2nd node (not an exit node) I have to manually go into vtysh and type router bgp 65000 followed by import vrf vrf_main in both ipv4 and ipv6 address families. It might be my network or something but the primary exit node doesn't seem to work
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom