Integrating a Proxmox VE with an External SDN, Applicable?

iami · Feb 6, 2024

Hello Proxmoxers

I'm planning to use an external SDN such as NSX-T or ACI but I don't know if I've to utilize Proxmox SDN or not?
Also, I didn't find something related to this subject either with Proxmox SDN or FRR.
In general, what are the best practices to be followed with the same use case?

thx,

iami · Feb 11, 2024

any response, Guys!

spirit · Feb 11, 2024

Hi,

This should require nsx-t implementation on proxmox side (sdn have plugin architecture to create new plugin).

Maybe you could interconnect proxmox bgp-evpn sdn with nsx-t through bgp,

but to use real nsx-t this need to use nsx-t on proxmox node + integration of plugin.

with broadcom licensing change, it's not going to happen for nsx-T, I don't known too much how cisco aci is working.

Personnaly, at work, I'm using bgp-evpn , using my arista routers as exit-nodes as they support evpn natively.

iami · Feb 12, 2024

Thank you Spirit for your answer,
1- in your setup, how do you make arista routers as an exit node through the drop list? can you share please simple diagram explaining your setup.
2- plz, correct me if I misunderstood you, you mean Proxmox SDN has a plan to create new plugins such as nsx-t that's integration plugin.
3- nsx-t will not support KVM in the new release. So, what do you mean here: this need to use nsx-t on proxmox node + integration of plug ?

spirit · Feb 12, 2024

iami said:
Thank you Spirit for your answer,
1- in your setup, how do you make arista routers as an exit node through the drop list? can you share please simple diagram explaining your setup.

you just need to peers between thrugh bgp/evpn from you arista routers with the proxmox nodes, then the arista announce the default gw
https://www.arista.com/en/um-eos/eos-integrated-routing-and-bridging

here an example of configuration: (proxmox use frr router, like cumulus linux in this article)
https://www.jasonvanpatten.com/2018/11/15/cumulus-and-arista-evpn-configuration/

iami said:
2- plz, correct me if I misunderstood you, you mean Proxmox SDN has a plan to create new plugins such as nsx-t that's integration plugin.

no

iami said:
3- nsx-t will not support KVM in the new release. So, what do you mean here: this need to use nsx-t on proxmox node + integration of plug ?

It's not about supporting kvm. It's more about supporting the proxmox linux kernel

https://docs.vmware.com/en/VMware-N...tml#GUID-B48076FC-99C3-46C9-A27B-44BF74609972

I have found reference about rhel kernel only. It seem they have a custom kernel module + openvswitch

With all the broadcom shit , it quite possible that nsx-t will be available in vwmware package now. (As it seem than broadcom don't sell indiviual licenses)

And after that, it should require to program their vswitch when we create a new vm ip/mac through an new sdn plugin.

iami · Feb 13, 2024

let me explain the first draft:

__________1__________ ------ __________2_________ ------ ________3________
| Proxmox Cluster | => | Controller Cluster | => | External SDN |

1- this cluster running the workload
2- this will be as a router and internal SDN for #1 (I will use proxmox here also)
3- this what I have to integrate with, mostly NSX-T

What does it look like?
In this case, do I have to use SDN or FRR for more options?

thanks,

spirit · Feb 13, 2024

if you already have an nsx-T sdn, I think the most easy way could be to do a bgp-evpn sdn on proxmox,

then find a way to exchange routes between both sdn through bgp

I never have used nsx-t, but looking at this article,

https://vstellar.com/2020/08/nsx-t-3-0-series-part-5-configure-logical-routing/

It's possible to do bgp on the nsx-edge node. (and on the proxmox side, on the exit-node)

iami · Feb 13, 2024

let's assume the following:
- in bgp-evpn, exit node is pve-01
- enabling bgp protocol between NSXT and pve-01
is that correct?
Also, Proxmox SDN will overwrite on pve-01's FRR config. how to handle that? especially, I need to add to add/remove vnets and subnets

spirit · Feb 13, 2024

iami said:
let's assume the following:
- in bgp-evpn, exit node is pve-01
- enabling bgp protocol between NSXT and pve-01
is that correct?

yes

iami said:
Also, Proxmox SDN will overwrite on pve-01's FRR config. how to handle that? especially, I need to add to add/remove vnets and subnets

you can configure bgp in proxmox sdn controllers configuration.
(add an extra bgp controller for pve-01)

iami · Feb 13, 2024

not working

YAML:

evpn: evpn
        asn 65004
        peers 192.168.100.105

bgp: bgppve01
        asn 65014
        node pve01
        peers 192.168.100.109,192.168.100.110
        bgp-multipath-as-path-relax 1
        ebgp 1
        ebgp-multihop 7

pve01 (exit node) = 192.168.100.105
nsx-t IPs = 192.168.100.109,192.168.100.110 (configured with asn=65004 )
pve02 (workload) = 192.168.100.106
pve03 (workload) = 192.168.100.107

also, I don't know how to add pve02 and pve03

spirit · Feb 13, 2024

iami said:
not working

YAML:

evpn: evpn asn 65004 peers 192.168.100.105 bgp: bgppve01 asn 65014 node pve01 peers 192.168.100.109,192.168.100.110 bgp-multipath-as-path-relax 1 ebgp 1 ebgp-multihop 7

pve01 (exit node) = 192.168.100.105
nsx-t IPs = 192.168.100.109,192.168.100.110 (configured with asn=65004 )
pve02 (workload) = 192.168.100.106
pve03 (workload) = 192.168.100.107

also, I don't know how to add pve02 and pve03

you don't need to define pve02/pve03, pve01 is the exit-node for the whole evpn network.
for you evpn controller, you need to have pve01,pve02,pve03 ip in peers, and try to use a different asn than 65004 (as it's already used by nsx-t)

can you share your /etc/pve/sdn/zones.cfg ?

you can also enable "advertise subnet" option in the evpn zone.

on pve-01, what is the result of:

# vtysh -c "sh ip bgp summary"

# vtysh -c "sh ip bgp"

?

iami · Feb 13, 2024

I've made some changes after the last post, let's start from here:

YAML:

evpn: nsxtzone
        controller nsxtbgp
        vrf-vxlan 11005
        advertise-subnets 1
        exitnodes pve01
        exitnodes-primary pve01
        ipam pve
        mac BC:24:11:22:23:24
        nodes pve02,pve03

spirit said:
you can also enable "advertise subnet" option in the evpn zone.

yes, it's enabled

Bash:

# vtysh -c "sh ip bgp summary"
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.100.105, local AS number 65014 vrf-id 0
BGP table version 4
RIB entries 7, using 1344 bytes of memory
Peers 2, using 1449 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor                        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
nsxt-t0-cluster-1(192.168.100.109) 4      65004       244       242        0    0    0 00:11:55            4        4 N/A
nsxt-t0-cluster-2(192.168.100.110) 4      65004       244       242        0    0    0 00:11:55            4        4 N/A

Total number of neighbors 2

Bash:

# vtysh -c "sh ip bgp"
BGP table version is 4, local router ID is 192.168.100.105, vrf id 0
Default local pref 100, local AS 65014
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 192.168.100.0/24  192.168.100.110(nsxt-t0-cluster-2)
                                             0             0 65004 ?
 *                   192.168.100.109(nsxt-t0-cluster-1)
                                             0             0 65004 65004 65004 65004 ?
 *> 192.168.110.0/24 192.168.100.110(nsxt-t0-cluster-2)
                                             0             0 65004 ?
 *                   192.168.100.109(nsxt-t0-cluster-1)
                                             0             0 65004 65004 65004 65004 ?
 *> 192.168.200.0/24 192.168.100.110(nsxt-t0-cluster-2)
                                             0             0 65004 ?
 *                   192.168.100.109(nsxt-t0-cluster-1)
                                             0             0 65004 65004 65004 65004 ?
 *> 192.168.241.0/24 192.168.100.110(nsxt-t0-cluster-2)
                                             0             0 65004 ?
 *                   192.168.100.109(nsxt-t0-cluster-1)
                                             0             0 65004 65004 65004 65004 ?

Displayed  4 routes and 8 total paths

spirit · Feb 13, 2024

so, it seem than you correctly receive nsx target routes.

are you able to check the same on nsx-t side ? (do you see proxmox vnets subnets ?)

also, in zone.cfg:

"nodes pve02,pve03"

try to add "nodes pve01,pve02,pve03" (or remove nodes filtering totally).
I don't remember if the exit-node configuration is correctly generated if the node itself is not in the list.

iami · Feb 14, 2024

yes, but all VMs running on pve02 & pve03 cannot reaching other VMs either on pve01 or external subnets through NSX-T

spirit · Feb 14, 2024

you need to fix first the communication between vms on the proxmox cluster

to see evpn routes on proxmox nodes

vtysh -c "sh bgp l2vpn evpn"

it should display mac && ip of the vms.

can you share your /etc/pve/sdn/subnets.cfg && /etc/pve/sdn/vnets.cfg too ?

iami · Feb 14, 2024

there's a connection between the VMs but it's gone once I enabled bgp controller on the exit node (pve01) and the connection comes back if I put the same asn for both evpn and bgp controllers

that's must be wrong

spirit · Feb 14, 2024

iami said:
there's a connection between the VMs but it's gone once I enabled bgp controller on the exit node (pve01) and the connection comes back if I put the same asn for both evpn and bgp controllers that's must be wrong

do you use same subnets on proxmox than vmware nsx ? (because It'll not work)

can you share your /etc/pve/sdn/subnets.cfg && /etc/pve/sdn/vnets.cfg too ?

iami · Feb 15, 2024

Yes, I'm talking about the subnets inside Proxmox, the external subnets are another challenges

Bash:

# cat /etc/pve/sdn/subnets.cfg /etc/pve/sdn/vnets.cfg
subnet: evpn-10.0.1.0-24
        vnet vnet01
        gateway 10.0.1.1

subnet: evpn-10.0.5.0-24
        vnet vnet01
        gateway 10.0.5.1

vnet: vnet01
        zone evpn
        tag 15000

spirit · Feb 15, 2024

In the controller.cfg,

can you try to use same 65014 asn for evpn && bgp

and in the bgppve01 bgp controller, add ips of all proxmox nodes + nsx-t
peers 192.168.100.109,192.168.100.110

evpn: evpn
asn 65014
peers 192.168.100.105,192.168.100.106,192.168.100.107

bgp: bgppve01
asn 65014
node pve01
peers 192.168.100.105,192.168.100.106,192.168.100.107,192.168.100.109,192.168.100.110
bgp-multipath-as-path-relax 1
ebgp 1
ebgp-multihop 7
[/CODE]

iami · Feb 15, 2024

not working!.
Also, it doesn't make sense to use the same ASN with eBGP and NSX-T ASN (65004) is required to define NSX-T as a remote peer

Integrating a Proxmox VE with an External SDN, Applicable?

New Member

New Member

Distinguished Member

New Member

Distinguished Member

New Member

Distinguished Member

New Member

Distinguished Member

New Member

Distinguished Member

New Member

Distinguished Member

New Member

Distinguished Member

New Member

Distinguished Member

New Member

Distinguished Member

New Member