Integrating a Proxmox VE with an External SDN, Applicable?

I

iami

New Member
Feb 6, 2024
14
0
1
Hello Proxmoxers :cool:

I'm planning to use an external SDN such as NSX-T or ACI but I don't know if I've to utilize Proxmox SDN or not?
Also, I didn't find something related to this subject either with Proxmox SDN or FRR.
In general, what are the best practices to be followed with the same use case?

thx,
 
Hi,

This should require nsx-t implementation on proxmox side (sdn have plugin architecture to create new plugin).

Maybe you could interconnect proxmox bgp-evpn sdn with nsx-t through bgp,

but to use real nsx-t this need to use nsx-t on proxmox node + integration of plugin.

with broadcom licensing change, it's not going to happen for nsx-T, I don't known too much how cisco aci is working.


Personnaly, at work, I'm using bgp-evpn , using my arista routers as exit-nodes as they support evpn natively.
 
Thank you Spirit for your answer,
1- in your setup, how do you make arista routers as an exit node through the drop list? can you share please simple diagram explaining your setup.
2- plz, correct me if I misunderstood you, you mean Proxmox SDN has a plan to create new plugins such as nsx-t that's integration plugin.
3- nsx-t will not support KVM in the new release. So, what do you mean here: this need to use nsx-t on proxmox node + integration of plug ?
 
iami said:
Thank you Spirit for your answer,
1- in your setup, how do you make arista routers as an exit node through the drop list? can you share please simple diagram explaining your setup.
Click to expand...
you just need to peers between thrugh bgp/evpn from you arista routers with the proxmox nodes, then the arista announce the default gw
https://www.arista.com/en/um-eos/eos-integrated-routing-and-bridging

here an example of configuration: (proxmox use frr router, like cumulus linux in this article)
https://www.jasonvanpatten.com/2018/11/15/cumulus-and-arista-evpn-configuration/
iami said:
2- plz, correct me if I misunderstood you, you mean Proxmox SDN has a plan to create new plugins such as nsx-t that's integration plugin.
Click to expand...
no ;)
iami said:
3- nsx-t will not support KVM in the new release. So, what do you mean here: this need to use nsx-t on proxmox node + integration of plug ?
Click to expand...
It's not about supporting kvm. It's more about supporting the proxmox linux kernel

https://docs.vmware.com/en/VMware-N...tml#GUID-B48076FC-99C3-46C9-A27B-44BF74609972

I have found reference about rhel kernel only. It seem they have a custom kernel module + openvswitch

With all the broadcom shit , it quite possible that nsx-t will be available in vwmware package now. (As it seem than broadcom don't sell indiviual licenses)

And after that, it should require to program their vswitch when we create a new vm ip/mac through an new sdn plugin.
 
Last edited:
let me explain the first draft:

__________1__________ ------ __________2_________ ------ ________3________
| Proxmox Cluster | => | Controller Cluster | => | External SDN |


1- this cluster running the workload
2- this will be as a router and internal SDN for #1 (I will use proxmox here also)
3- this what I have to integrate with, mostly NSX-T

What does it look like?
In this case, do I have to use SDN or FRR for more options?


thanks,
 
Last edited:
let's assume the following:
- in bgp-evpn, exit node is pve-01
- enabling bgp protocol between NSXT and pve-01
is that correct?
Also, Proxmox SDN will overwrite on pve-01's FRR config. how to handle that? especially, I need to add to add/remove vnets and subnets
 
iami said:
let's assume the following:
- in bgp-evpn, exit node is pve-01
- enabling bgp protocol between NSXT and pve-01
is that correct?
Click to expand...
yes
iami said:
Also, Proxmox SDN will overwrite on pve-01's FRR config. how to handle that? especially, I need to add to add/remove vnets and subnets
Click to expand...
you can configure bgp in proxmox sdn controllers configuration.
(add an extra bgp controller for pve-01)
 
not working :(

YAML: 
evpn: evpn
        asn 65004
        peers 192.168.100.105

bgp: bgppve01
        asn 65014
        node pve01
        peers 192.168.100.109,192.168.100.110
        bgp-multipath-as-path-relax 1
        ebgp 1
        ebgp-multihop 7

pve01 (exit node) = 192.168.100.105
nsx-t IPs = 192.168.100.109,192.168.100.110 (configured with asn=65004 )
pve02 (workload) = 192.168.100.106
pve03 (workload) = 192.168.100.107

also, I don't know how to add pve02 and pve03
 
iami said:
not working :(

YAML: 
evpn: evpn
        asn 65004
        peers 192.168.100.105

bgp: bgppve01
        asn 65014
        node pve01
        peers 192.168.100.109,192.168.100.110
        bgp-multipath-as-path-relax 1
        ebgp 1
        ebgp-multihop 7

pve01 (exit node) = 192.168.100.105
nsx-t IPs = 192.168.100.109,192.168.100.110 (configured with asn=65004 )
pve02 (workload) = 192.168.100.106
pve03 (workload) = 192.168.100.107

also, I don't know how to add pve02 and pve03
Click to expand...

you don't need to define pve02/pve03, pve01 is the exit-node for the whole evpn network.
for you evpn controller, you need to have pve01,pve02,pve03 ip in peers, and try to use a different asn than 65004 (as it's already used by nsx-t)


can you share your /etc/pve/sdn/zones.cfg ?

you can also enable "advertise subnet" option in the evpn zone.


on pve-01, what is the result of:

# vtysh -c "sh ip bgp summary"

# vtysh -c "sh ip bgp"

?
 
I've made some changes after the last post, let's start from here:

YAML: 
evpn: nsxtzone
        controller nsxtbgp
        vrf-vxlan 11005
        advertise-subnets 1
        exitnodes pve01
        exitnodes-primary pve01
        ipam pve
        mac BC:24:11:22:23:24
        nodes pve02,pve03

spirit said:
you can also enable "advertise subnet" option in the evpn zone.
Click to expand...
yes, it's enabled

Bash: 
# vtysh -c "sh ip bgp summary"
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.100.105, local AS number 65014 vrf-id 0
BGP table version 4
RIB entries 7, using 1344 bytes of memory
Peers 2, using 1449 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor                        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
nsxt-t0-cluster-1(192.168.100.109) 4      65004       244       242        0    0    0 00:11:55            4        4 N/A
nsxt-t0-cluster-2(192.168.100.110) 4      65004       244       242        0    0    0 00:11:55            4        4 N/A

Total number of neighbors 2

Bash: 
# vtysh -c "sh ip bgp"
BGP table version is 4, local router ID is 192.168.100.105, vrf id 0
Default local pref 100, local AS 65014
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 192.168.100.0/24  192.168.100.110(nsxt-t0-cluster-2)
                                             0             0 65004 ?
 *                   192.168.100.109(nsxt-t0-cluster-1)
                                             0             0 65004 65004 65004 65004 ?
 *> 192.168.110.0/24 192.168.100.110(nsxt-t0-cluster-2)
                                             0             0 65004 ?
 *                   192.168.100.109(nsxt-t0-cluster-1)
                                             0             0 65004 65004 65004 65004 ?
 *> 192.168.200.0/24 192.168.100.110(nsxt-t0-cluster-2)
                                             0             0 65004 ?
 *                   192.168.100.109(nsxt-t0-cluster-1)
                                             0             0 65004 65004 65004 65004 ?
 *> 192.168.241.0/24 192.168.100.110(nsxt-t0-cluster-2)
                                             0             0 65004 ?
 *                   192.168.100.109(nsxt-t0-cluster-1)
                                             0             0 65004 65004 65004 65004 ?

Displayed  4 routes and 8 total paths
 
so, it seem than you correctly receive nsx target routes.

are you able to check the same on nsx-t side ? (do you see proxmox vnets subnets ?)


also, in zone.cfg:

"nodes pve02,pve03"

try to add "nodes pve01,pve02,pve03" (or remove nodes filtering totally).
I don't remember if the exit-node configuration is correctly generated if the node itself is not in the list.
 
yes, but all VMs running on pve02 & pve03 cannot reaching other VMs either on pve01 or external subnets through NSX-T
 
Last edited:
you need to fix first the communication between vms on the proxmox cluster

to see evpn routes on proxmox nodes

vtysh -c "sh bgp l2vpn evpn"

it should display mac && ip of the vms.

can you share your /etc/pve/sdn/subnets.cfg && /etc/pve/sdn/vnets.cfg too ?
 
there's a connection between the VMs but it's gone once I enabled bgp controller on the exit node (pve01) and the connection comes back if I put the same asn for both evpn and bgp controllers :( that's must be wrong
 
iami said:
there's a connection between the VMs but it's gone once I enabled bgp controller on the exit node (pve01) and the connection comes back if I put the same asn for both evpn and bgp controllers :( that's must be wrong
Click to expand...
do you use same subnets on proxmox than vmware nsx ? (because It'll not work)

can you share your /etc/pve/sdn/subnets.cfg && /etc/pve/sdn/vnets.cfg too ?
 
Yes, I'm talking about the subnets inside Proxmox, the external subnets are another challenges

Bash: 
# cat /etc/pve/sdn/subnets.cfg /etc/pve/sdn/vnets.cfg
subnet: evpn-10.0.1.0-24
        vnet vnet01
        gateway 10.0.1.1

subnet: evpn-10.0.5.0-24
        vnet vnet01
        gateway 10.0.5.1

vnet: vnet01
        zone evpn
        tag 15000
 
In the controller.cfg,

can you try to use same 65014 asn for evpn && bgp

and in the bgppve01 bgp controller, add ips of all proxmox nodes + nsx-t
peers 192.168.100.109,192.168.100.110


evpn: evpn
asn 65014
peers 192.168.100.105,192.168.100.106,192.168.100.107

bgp: bgppve01
asn 65014
node pve01
peers 192.168.100.105,192.168.100.106,192.168.100.107,192.168.100.109,192.168.100.110
bgp-multipath-as-path-relax 1
ebgp 1
ebgp-multihop 7
[/CODE]
Click to expand...
 
not working!.
Also, it doesn't make sense to use the same ASN with eBGP and NSX-T ASN (65004) is required to define NSX-T as a remote peer
 
You must log in or register to reply here.
Top Bottom