Evpn SDN with NAT-like routing

[Mephisto aD]

Hi all,

I recently discovered the SDN feature of the Proxmox Virtual Environment.
So far it looks great for separating all guests into different networks, but I ran into some difficulties / questions:

So far https connections to given domains, neither IPs outside the local network (containing the pve servers), are possible (I tried this with various guests in the network), but http connections for example are working. However the DNS resolution works, so I assume it's a routing problem...

This is my current SDN config:

# controllers.cfg
evpn: standard
  asn 64513
  peers 192.168.2.11,192.168.2.12
 
# zones.cfg
evpn: overlay
    controller standard
    vrf-vxlan 12345
    advertise-subnets 1
    exitnodes eris
    ipam pve
    mac AE:96:34:54:85:F4
   
# vnets.cfg
vnet: test
    zone overlay
    tag 123456
   
# subnets.cfg
subnet: overlay-10.0.0.0-24
    vnet test
    gateway 10.0.0.1
    snat 1

Aditionally I'd like to allow the access from certain subnets on the hosts from other subnets (but not their gateway) is there a way to cunfigure this already the proxmox way?

Thanks a lot for any answer
Wish you all a happy new year

 

[spirit]

So far https connections to given domains, neither IPs outside the local network (containing the pve servers), are possible (I tried this with various guests in the network), but http connections for example are working.

so http is working, but not https ? It could be a mtu problem with fragmented packet

Aditionally I'd like to allow the access from certain subnets on the hosts from other subnets (but not their gateway) is there a way to cunfigure this already the proxmox way?

are you talking about subnets in the evpn network ? because they can communicate in the same zone through the anycast gateway defined on the subnet, and you can't connect to the gateway (ssh or other protocol), because this gateway is an specific vrf.

 

[Mephisto aD]

Heyy, thanks for your answer

It could be a mtu problem with fragmented packet

I randomly set the MTU of the zone to 16000 (just to test it once), but sadly it still doesn't work... So I guess I can exclude this option :-/

because they can communicate in the same zone through the anycast gateway defined on the subnet, and you can't connect to the gateway (ssh or other protocol), because this gateway is an specific vrf.

Oh, great! I didn't notice this so far
Is there a way to partially disable this function? I mean I'd like to have for example one (evpn) network called admin from where I could access the other networks, but without the other systems being capable of acessing the admin systems... Or do I have to route the traffic between different zones then?
As far as I saw that at least from the gui this is not possible, right?

 

[spirit]

Heyy, thanks for your answer

I randomly set the MTU of the zone to 16000 (just to test it once), but sadly it still doesn't work... So I guess I can exclude this option :-/

The original mtu, is set where the source ip if defined.
For example, if your vm have mtu 1500 inside the guest os, mtu 1500 will be used.
When it'll be encapsulated through the vxlan, it'll add 50 bytes for vxlan, so mtu 1550.

for the bridge zone inside the host (mtu=16000), it's ok, it'll allow to forward it without fragmentation.
now, what about your physical switches ? do you have configured jumbo frame/ allow bigger mtu than 1500 ?
Same if you have physical routers,etc...

If not, you need to lower mtu inside your vm guest os. (last proxmox version also allow to change it in the vm nic gui)

Oh, great! I didn't notice this so far
Is there a way to partially disable this function? I mean I'd like to have for example one (evpn) network called admin from where I could access the other networks, but without the other systems being capable of acessing the admin systems... Or do I have to route the traffic between different zones then?
As far as I saw that at least from the gui this is not possible, right?

Well, from the network routing view, if your admin network have access to your other network, the other network will have access to your admin network. (This is pure routing, you can't known flow direction). The best way is to use proxmox firewall.


What is could be possible (not implemented yet, with route leak/import), could be to create multiple zones like

1 zone: admin
1 zone: customer1
1 zone: customer2

Where admin could have access to customer1 && customer2 (and customer1/2 have access to admin), but customer1 don't have access to customer2.

 

[Mephisto aD]

If not, you need to lower mtu inside your vm guest os. (last proxmox version also allow to change it in the vm nic gui)

Hmm this didn't work either for me. Not sure if it's the MTU at all...

What is could be possible (not implemented yet, with route leak/import), could be to create multiple zones like

I just tried it again and found the field `Route-target-import` which in the gui for "zones" is described in the documentation as "Allows you to import a list of external EVPN route targets. Used for cross-DC or different EVPN network interconnects."

Could this be the import option you meant? and in case it is, do you know what values to put into this field?

 

[spirit]

Hmm this didn't work either for me. Not sure if it's the MTU at all...

Just try a: "ping -s 1470 -Mdo <targetip>". -Mdo = do not fragment. If you have an mtu problem, the ping should tell you

I just tried it again and found the field `Route-target-import` which in the gui for "zones" is described in the documentation as "Allows you to import a list of external EVPN route targets. Used for cross-DC or different EVPN network interconnects."

Could this be the import option you meant? and in case it is, do you know what values to put into this field?

Not related. This is if you have differents evpn networks (with differents AS, differents proxmox clusters), and you want to be able to communicate between them.

 

[Mephisto aD]

Just try a: "ping -s 1470 -Mdo <targetip>". -Mdo = do not fragment. If you have an mtu problem, the ping should tell you

okay, Yeah, you have been right, this didn't work either, so it was probably the mtu.

Now I reconfigured my sdn to create the two zones and try a route leaking between them. You might laugh now, but now the mtu bug is gone, and I have honestly no clue why:

root@haumea:/etc/pve/sdn# cat zones.cfg
evpn: infra
    controller standard
    vrf-vxlan 10000
    exitnodes haumea,gaia,eris
    ipam pve
    mac AE:96:34:54:85:F4
    mtu 1450   # this mtu was suggested in the documentation, but in the testing zone it also works without it

evpn: testing
    controller standard
    vrf-vxlan 10001
    advertise-subnets 1
    exitnodes haumea,gaia,eris
    ipam pve
    mac 3A:8B:BD:8F:05:A2

root@haumea:/etc/pve/sdn# cat vnets.cfg
vnet: testing
    zone testing
    tag 1000100

vnet: infra
    zone infra
    tag 1000000

root@haumea:/etc/pve/sdn# cat subnets.cfg
subnet: infray-10.0.0.0-24
    vnet infra
    gateway 10.0.0.1
    snat 1

subnet: overlay-10.1.0.0-24
    vnet testing
    gateway 10.1.0.1
    snat 1

now a simple `curl https://www.google.com` works from a lxc inside each of this networks.
Additionally routing between both networks seems to work by default. (At least I can make HTTP requests from one to another.)

Is there any way to limit this? The following doesn't seem to work at least...

root@haumea:/etc/pve/firewall# cat cluster.fw
[OPTIONS]

enable: 1

[IPSET infra]

10.0.0.0/24

[IPSET testing]

10.1.0.0/24

[RULES]

IN REJECT -i testing -source +testing -dest +infra -p udp -log nolog
IN REJECT -i testing -source +testing -dest +infra -p tcp -log nolog