SDN Configuration Question

B

Brandon

Active Member
Feb 1, 2017
10
0
41
32
Hi,

I'm working on setting up SDN in a new cluster. The first question is about the physical network topology.

Option 1)

Code: 
---------
-- ISP --
---------
   |
   V
-----------------------
-- OpnSense Firewall --
-----------------------
       |
       V
----------------------------
-- Proxmox Hosts With SDN --
--   3 hosts in cluster   --
----------------------------
             |
             V
---------------------------
--     LXC Containers    --
---------------------------

Or is it possible to skip the OPNSense firewall at the edge and provision public IP addresses into the vxlans?

Option 2)

Code: 
---------
-- ISP --
---------
   |
   V
----------------------------
-- Proxmox Hosts With SDN --
--   3 hosts in cluster   --
----------------------------
             |
             V
--------------------------------------------
--          OpnSense Firewall VM          --
-- Wan bridged to Wan adapter on the host --
-- Lan mapped to the vxlan zone/vnet      --
--------------------------------------------
                 |
                 V
------------------------------------
--          LXC Containers        --
--  Lan mapped to vxlan zone/vnet --
------------------------------------

Option 1 is the standard setup for most companies, but it doesn't give you isolation between the containers on the network. Option 2 gives you more control to each of the container networks that are isolated, but it requires mapping the wan on the opnsense vm to the vmbr0 of a specific node. Option 2 is not ideal, because you wouldn't be able to live migrate the opnsense VM to another node without the whole network going down.

The other option I've been looking at is evpn, but I haven't been able to wrap my head around how the it should work and how to assign public IPs from our ISP to the containers/vms.

Any insight is much appreciated.

Thanks
 
Hi,

Personnaly, at work, I'm doing evpn with public ip inside my vm + proxmox firewall.

for evpn, you need to define exit-nodes. They will route traffic between your evpn/vxlan network and your isp.
and your isp need to route traffic to your exit-nodes.


I have also a customer, doing evpn, with a pair of pfsense server between the isp && the evpn network for firewalling + nat (private network in the vms). pfsense nodes are routing to evpn exit-nodes. (as pfsense can't do evpn natively. if one day pfsense is able to do evpn, you could use pfsense as exit-node directly).



without evpn, you have only layer2 vxlan, so you need something like pfsense like on your schemas, to route between vxlan && real network.


for your option2, can't you have 2 pfsense vms with a carp vip ?
 
Thanks for the reply Spirit.

Personnaly, at work, I'm doing evpn with public ip inside my vm + proxmox firewall.
Click to expand...

So you have isp directly into pve host. Configured evpn with exit nodes. Then on your vm you assigned a public IP on the vm. Do you then have a second nic attached to the same evpn zone for routing subnet traffic (kind of similar to option 2 above)?

Here is what I have setup:
Code: 
---------
-- ISP --
---------
    |
    V
------------------------------------------------
-- port1 on physical host1 connected to isp   --
-- vmbr10 (no ip) bridge port1                --
-- port2 on pve host connected to lan switch  --
-- vmbr20 (no ip) bridge port2 for lan        --
-- port3 on pve host connected to lan switch  --
-- vmbr0 (pve ip address on lan) bridge port3 --
------------------------------------------------
      |
      V
-------------------------
-- Opnsense VM         --
-- vmbr10 (wan bridge) --
-- vmbr20 (lan bridge) --
-------------------------
        |
        V
------------------------------------------------
-- lan network where containers are currently --
------------------------------------------------

I setup evpn controller with an exit node being the physical host that has the wan interface connected for opnsense currently. But I think i'm running into an issue with routing across virtual network bridges since the host is configured with a default gateway that is the VM that is running on vmbr10.

Here is the route:

Code: 
default via 10.100.11.1 dev vmbr0 proto kernel onlink
10.10.10.0/24 nhid 91 via 10.255.255.2 dev xvrf_evpn proto static metric 20
10.100.11.0/24 dev vmbr0 proto kernel scope link src 10.100.11.150
10.255.255.0/30 dev xvrf_evpn proto kernel scope link src 10.255.255.1

I can ping between 2 machines attached to the zone/vnet (10.10.10.0/24) for evpn. I can also ping the ip of the exit node (10.100.11.150/24). Then from there I cannot ping out to anything else on the lan or public internet.

I also noticed connectivity issues when PVE firewall is turned on at the pve host level (i have disabled them temporarily now to test sdn).

Here is the SDN configs if it helps:

Code: 
evpn: evpn
        asn 65000
        peers 10.100.11.150,10.100.11.155,10.100.11.157

subnet: evpn-10.10.10.0-24
        vnet asdf
        gateway 10.10.10.1
        snat 1

vnet: asdf
        zone evpn
        tag 2

evpn: evpn
        controller evpn
        vrf-vxlan 1
        advertise-subnets 1
        exitnodes backup1
        exitnodes-local-routing 1
        ipam pve
        mac 26:A3:E5:F9:55:C1

Thanks in advanced for the help.
 
I have also tried toggling on and off snat at the vnet and it didn't seem to make any difference here.
 
So you have isp directly into pve host. Configured evpn with exit nodes. Then on your vm you assigned a public IP on the vm. Do you then have a second nic attached to the same evpn zone for routing subnet traffic (kind of similar to option 2 above)?
Click to expand...

no, in pure evpn, you don't need any router vm.
what you can do: (with public ip in the vm, no nat, but in this example I'll use private ip for the demo):

for incoming traffic

the isp router always send traffic to the exit-node, then then exit-node forward traffic to the vm (which can be located on any nodes)


Code: 
isp router (192.168.0.254)----------------->eth0(192.168.0.1)  proxmox exit-node------>vnet1 (10.0.0.254)--------------->  vm1 (10.0.0.1)
  (route: 10.0.0.0/8 gw 192.168.0.1)

for outgoing traffic

Code: 
vm1(10.0.0.1)------------------------->vnet1(10.0.0.254)----------->proxmox exit-node eth0(192.168.0.1)------------------> isp router (192.168.0.254)
(default gw: 10.0.0.254)                                                                      (default gw 192.168.0.254)


your vm gw is the vnet ip. (This is an anycast ip, this same gw ip exist on everywhere nodes).

Then from, the vnet, the traffic to outside (0.0.0.0) is forwarded to the exit-node.

then the exit-node node is forwarding traffic between evpn network and your isp router through is default gw.




Now, if you want add to add some pfsense for nat+firewall, what you can do:


Code: 
isp router (192.168.0.254)-----------------> (192.168.0.1)pfsense eth0(172.16.0.254)--------->(172.16.0.1)proxmox exit-node------>vnet1 (10.0.0.254)--------------->  vm1 (10.0.0.1)
  (route: 10.0.0.0/8 gw 192.168.0.1)                        (route 10.0.0.0/8 gw 172.16.0.1)



I also noticed connectivity issues when PVE firewall is turned on at the pve host level (i have disabled them temporarily now to test sdn).
Click to expand...

yes, evpn/vxlan use udp/4789 between nodes, this need to be open.
 
We're making progress.

I can now ping the following:

Code: 
LXC CT ---> vnet gateway
LXC CT ---> exit node IP
LXC CT ---> firewall/router lan ip
LXC CT ---> lan network ips
LXC CT ---> public internet
firewall/router ---> LXC CT
exit node ip ---> LXC CT
lan network ips ---> LXC CT

The weird part now is that the container can ping essentially everything, but i cannot get DNS to resolve anything from within the CT.

Code: 
nslookup google.com 8.8.8.8
or
nslookup google.com <pfsense lan ip>

neither of them complete.

I thought it was going to be a MTU problem, but the host machine autoconfigured the vrf interfaces as 1450.

Code: 
96: asdf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpn state UP group default qlen 1000
97: vrf_evpn: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000
99: vrfbr_evpn: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpn state UP group default qlen 1000
114: vxlan_asdf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master asdf state UNKNOWN group default qlen 1000
115: vrfvx_evpn: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrfbr_evpn state UNKNOWN group default qlen 1000

Here are the adapters to the bridge interfaces (including the virtual firewall).

Code: 
2: enp2s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr0 state DOWN group default qlen 1000
3: enp2s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr0 state DOWN group default qlen 1000
6: enp132s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr10 state UP group default qlen 1000
7: enp132s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr20 state UP group default qlen 1000
8: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.100.11.150/24 scope global vmbr0
10: vmbr10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
11: vmbr20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
80: fwpr120p0@fwln120i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr20 state UP group default qlen 1000
84: fwpr120p1@fwln120i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr10 state UP group default qlen 1000

The other strange thing is, I can ssh from machines on the Lan to the the CT ip and vice versa. That tells me that TCP traffic should be able to reach the container and back. I'm kind of running out of ideas.
 
The other thing i'm seeing is that on reboot of the container it is starting the container with an internal mtu of 1500:

Code: 
root@test-net2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if110: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:18:48:30:44:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.11.11.11/24 brd 10.11.11.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ac18:48ff:fe30:44a1/64 scope link tentative
       valid_lft forever preferred_lft forever
root@test-net2:~# tracepath -4 -n 8.8.8.8
 1?: [LOCALHOST]                      pmtu 1500
^C

But if i tracepath with mtu at 1450 manually or change it via the ip link set eth0 mtu 1450 command it does tracepath correctly.

Code: 
root@test-net2:~# tracepath -4 -n 8.8.8.8 -l 1450  
 1:  10.11.11.1                                            0.074ms  
 2:  no reply
 3:  10.100.11.1                                           1.200ms asymm  4  
 4:  <REDACTED>                                       2.225ms asymm  5  
 5:  <REDACTED>                                        5.973ms asymm  6  
 6:  <REDACTED>                                       31.858ms asymm 12
 
indeed, you need to lower mtu in your guest os too. It's no yet automatically done by proxmox.
Click to expand...

If the exitnode is 1500, should the evpn zone be set 1450 and the guest manually set to 1450?
 
the isp router always send traffic to the exit-node, then then exit-node forward traffic to the vm (which can be located on any nodes)
Click to expand...

what if pings stop on exit node which is always the Proxmox Host
 
eset said:
what if pings stop on exit node which is always the Proxmox Host
Click to expand...

That is probably SNAT on the vnet. I have noticed that SNAT doesn't always reload with apply (ifreload -a). I traced packets to the firewall and if you disable the SNAT option and apply it doesn't always work. The packets that reach the firewall show the IP of the exit node and do not go back to to the subnet addresses. The only way I was able to get SNAT to disable was to power off the CT and then delete the entire zone and recreate it.
 
Hi spirit,

I'm working on setting up SDN in a new cluster and looking the way to assign public IP to VM. My Proxmox exit-node connected with ISP via L2 bridge. Take a note IPS router, exit-node and vm1 has same /24 subnet IP.

I would appreciate any help

Code: 
isp router (5.133.66.1)----------------->vmbr0(5.133.66.21)  proxmox exit-node------>vnet1 (?.?.?.?)--------------->  vm1 (5.13.66.105)
 
Hi,
I'm not sure, but maybe you can reuse 5.133.66.1 for vnet1. (It's a different vrf, it should not conflict).

and on your isp router, try to add a route like:

5.133.66.105/32 gw 5.133.66.21



maybe better :

do you have access to is p router ? if yes, can you add additional private network ?

for example:

Code: 
(5.133.66.1)<---- isp router (10.0.0.1)----------------->vmbr0(10.0.0.254)  proxmox exit-node------>vnet1 (5.133.66.1)--------------->  vm1 (5.133.66.105)
route add 5.133.66.105/32 gw 10.0.0.254
 
Tried to reuse 5.133.66.1 but it does not work. At the moment I don't have access to isp router, willing to contact them to add private network as you suggested.
Thank you for your help spirit.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back