SDN, eVPV and Zones

[pille99]

hello all
i try to do following. as in the picture seen, i need to zones (one is the windows network with 172.15.101-110 and the other the Openshift Network with 172.1.199.100-110)
the windows network is only on server which need to connect to the internet and 3 to other servers in other zones. the same for the openshift network

i created 2 Controllers (lets call them A and B), 2 Zones (lets call them 1 and 2) and the dependent Vnets (lets call them ! and ?).

the chain is
A <> 1 <> !
and
B <> 2 <> ?

here are some questions

1. Multiple VNets vs single Vnet and multiple Subnets ?
means: Vnet win100, win101, win102, and so on and vnet OpnSft100, OpnSft101, OpnSft102, and so on
or Vnet WinvNet with subnets 100, 101, 102, and so on and for the Openshift the same

as iunderstood: the subnets in the vNets can communicate out of box to each other vs with multiple VNets i need a router (opnsense) between - correct ?
what is the best way to archive te challange ?

2. to connect to "outside" (and it would mean already between 2 different zones or to any other service (like a centralized DNS) i need an exit host - is that correct ?
but dont i leave than the "secure" environment ?

i want in front the the windows network an opnsense for firewalling (no traffic is allowed in and out / only DNS and 3 servers will connect to another Server in the public area)

3. in the econtroler config is ask for "peers" and in the evpn config ask for exit nodes - isnt that the same ?

thx for answers

 

[spirit]

1. Multiple VNets vs single Vnet and multiple Subnets ?
means: Vnet win100, win101, win102, and so on and vnet OpnSft100, OpnSft101, OpnSft102, and so on
or Vnet WinvNet with subnets 100, 101, 102, and so on and for the Openshift the same

each vnet is a different vxlan bridge. (like a different vlan if you want). So you have layer2 isolation here.

as iunderstood: the subnets in the vNets can communicate out of box to each other vs with multiple VNets i need a router (opnsense) between - correct ?
what is the best way to archive te challange ?

each subnet in the same zone are able to communicate through the anycast gateway (the gateway ip that you define on the subnet). No need opensense for the gateway of the vm. (This gateway ip exist on all proxmox nodes, that mean that when you migrate a vm, the vm always communicate with the local anycast gateway. Then the evpn network is forwarding traffic internally to other nodes if needed)

2. to connect to "outside" (and it would mean already between 2 different zones or to any other service (like a centralized DNS) i need an exit host - is that correct ?
but dont i leave than the "secure" environment ?

i want in front the the windows network an opnsense for firewalling (no traffic is allowed in and out / only DNS and 3 servers will connect to another Server in the public area)

if you opensense is a vm, you can setup opensense with 1 interface in vxlan network (vnet..) and 1 interface in real network (vmbrX) for example.

if you opensense is a physical server, the problem is that opensense can't do evpn natively currently. So you need to use exit-node as gateway between evpn and real network + bgp peers with opensense to route traffic to it.

In the econtroler config is ask for "peers" and in the evpn config ask for exit nodes - isnt that the same ?

no.

the peers is the list of ips of all proxmox nodes where you want the evpn network.

The exit-nodes can be only 1 or 2 nodes for example, and they are only use to forward traffic to real network.

 

[pille99]

hey
thx for your answer
i added the exit nodes and the gateway address in the subnets and servers from a can ping the gateway from subnet b, but not from server to server of 2 different subnets.

i didnt understand the part of the opnsense
lets say
subnet a
subnet b
subnet c
(in each subnet are 1 or 2 servers which need to cummunicate to outside)
you said: add the opnsense to vnet. would it be than a new ip ?

PS: i have an own network for the sdn traffic. 10.10.15.10-13. it should be only for the internal communication between servers on different nodes, which worked with vxvlan pretty well.

so. 2 points are not solved yet
1. communication between servers in different subnets
2. opnsense (which will forward for example the DNS querries to the corerct DNS server outside of the sdn network)

thx

 

[spirit]

i added the exit nodes and the gateway address in the subnets and servers from a can ping the gateway from subnet b, but not from server to server of 2 different subnets.

This is not normal. you should be able to ping servers in the same zone, if they are on differents vnets/subnet in this zone.

can you send your /etc/pve/sdn/*.cfg files ?

(in each subnet are 1 or 2 servers which need to cummunicate to outside)

so you need some s-nat somewhere for private -> public.

2 possibilities:

1)
if your exit-node proxmox host have a public ip address, you can enable "s-nat" option on the subnet, it'll do s-nat/masquerade on the exit node. (you also need to configure a primary exit node in the zone, as s-nat not working with balancing active/active between 2 exit-nodes)


2)
or you could have an external opensense doing the nat,
in this case you need to configuring routing between the exit-nodes && opensense.

-can be done with bgp for example. (you need to add an extra bgp controller for each exit-node, and add a peer with ip of opensense).
-or you can do static route:
on the exit-node host: add default gw to the opensense
on the opensense, add a route like : route add <yourevpnsubnets> gw <your-proxmox-exit-node-real-ip>



The network path is something like for example

vm ip (172.1.199.100)---------->(vnet subnet gateway)---------(internal evpn routing 0.0.0.0/0)----->exit-node-(proxmox host ip 10.10.15.10) ----(default gw)---------->10.10.15.254 opensense ((snat)-----> internet

and in reverse side

internet---> opensense- 10.10.15.254---> route 172.1.199.0/24--->(proxmox host ip 10.10.15.10)exit-node-----(internal evpn routing)-------->(vnet subnet gateway)---->vm ip (172.1.199.100)

 

[pille99]

thx once again
just to be clear - with "external" i mean outside of the eVPN.
the 10 network are pysically network cards which are connected on a switch. theses network doesnt have internet access. it soley for the SDN Traffic.

as i understand, to access the 10.10.15.x, i need to create a vLAN in top of the physically NIC. for all 4 Nodes. than i can access it with Opnsense.
i need to go over opnsense because i want active block in and outgoing traffik

my goal (just to define it) - each network and the containing VMs should communicate to each other (which proxmox cant, out of box). i move the VMs a bit and it just should work (with vxlan worked excellent. i just want to make it more secure and a better funktionality and performance.