SDN + netns: namespaces, bridging and VLANS

[cxor]

Mainly, it would be fantastic if someone could give me a "here be dragons" on using Proxmox SDN with policy-based routing (i.e. multiple gateways with iproute2 tables + `ip rule`).

I would like some more advanced networking (e.g. multiple NICs: i have will have 4 very soon). In configuring these, I would like to avoid the very difficult-to-troubleshoot issues that might occur if I combine Proxmox's approach to networking with my own approach in the wrong way.

The main issue I am trying to overcome is the lack of true routing with the Linux virtual devices and most of the simpler Proxmox SDN. I need a simple way to define `ip route` and `ip rule` configurations to use multiple routing tables so that I can have multiple default gateways.

- If i add content to /etc/iproute2/rt_tables, will proxmox autoupdate this?
- Are the sourced networking scripts and post-up/etc scripts true scripts? Or are there limitations to what they can do?
- what is Proxmox using to manage the virtual ethernet & bridge devices?

- How compatible would Proxmox SDN be with netns namespaced virtual ethernet devices? I have been thinking about lightly combining the techniques in this series of article with Proxmox SDN, but preferring 100% SDN where possible.

https://linux-blog.anracom.com/2018...ans-in-unnamed-linux-network-namespaces-viii/

To do this, I don't need namespaces, but they can make things simpler to define/isolate.

 

[spirit]

Hi,
the evpn zone already support already 1 vrf by zone. (vrf are better/cleaner than namespace).

But I think It could implement it for "simple" zone plugin. (simple zone are used for simple routed setup).
ifupdown2 already support "vrf ..." option on interfaces, but I don't known if gateway syntax is alrady using it.

- If i add content to /etc/iproute2/rt_tables, will proxmox autoupdate this?

proxmox don't touch this file

- Are the sourced networking scripts and post-up/etc scripts true scripts? Or are there limitations to what they can do?

yes, you can execute any command in post-up , pre-up,...

- what is Proxmox using to manage the virtual ethernet & bridge devices?

vm use tap interface, ct use veth interfaces.
They are plugged on a linux bridge (vmbrX) or openvswitch if you want to use it.


So putting the bridge in a vrf or namespace should be enough to be able to use multiple default gw.

 

[cxor]

merci beaucoup @spirit

 

[spirit]

Hi,
Seem to works with ifupdow2.

sample /etc/network/interfaces

auto eno1
iface eno1
        address 10.0.0.1/24
        gateway 10.0.0.254


auto vrf1
iface vrf1
        vrf-table auto

auto eno2
iface eno2
        address 192.168.0.1/24
        gateway 192.168.0.254
        vrf vrf1

# ip route show table all

default via 192.168.0.254 dev eno2 table vrf1 proto kernel onlink
default via 10.0.0.254 dev eno1 proto kernel onlink

so, maybe can you try to already add "vrf ..." on vmbrX too.

If you are able to get it work, just post the working config, I'll try to see to implement it in sdn plugin.

 

[cxor]

I had not heard of VRFs, so thank you!

So far I've got one VRF up and one guest VM using it. From the guest, I verified that traffic is being routed through the VRF by default with:

`traceroute -i dns.my.local`

Here is my current config:

iface enp1s0 inet manual
iface enp2s0 inet manual

auto vmbr1
iface vmbr1 inet static
    address 192.168.10.10/24
    gateway 192.168.10.1
    bridge-ports enp1s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4096
   
auto vrf2
iface vrf2
    vrf-table auto

auto vmbr2
iface vmbr2 inet static
    vrf vrf2
    address 192.168.20.10/24
    gateway 192.168.20.1
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4096

I installed a new NIC to give me a total of 4 ethernet ports. I will update the post after verifying further configuration with VRF's soon, including the bridge vmbr2 subdevices. I hope to configure two SDN VLANs, which connect through vmbr2 to into the router on the other side. It will require trunking/tagging and some VLAN changes on the router (i think)

I am in the middle of trying to set up Ansible to manage iptables on some routers in my network backbone area, so it may be a week or so before I can move on to further config.

 

[spirit]

. I hope to configure two SDN VLANs, which connect through vmbr2 to into the router on the other side. It will require trunking/tagging and some VLAN changes on the router (i think)

just to be sure, in your setup, what is the gateway of your vms ? (for example, in your example, is vmbr2 192.168.20.10 the gateway of your vms in 192.168.20.0/24 ?)

 

[cxor]

I have a Proxmox server with two network interfaces. I ordered a PCIe card and now I have four. I'm only using two though: one for a `10.10.0.0/16` network and one for a `10.20.0.0/16` network. (i'm changing the addressing a bit to provide more room).

In order to get the route summarization to work, I think a router is required to be in front of the interface. By doing this, I was hoping to make ACL and iptables management easier since rules can be written for the entire 10.10.0.0/16 network. However, my bridge gateways are not on separate networks.

just to be sure, in your setup, what is the gateway of your vms ?

To answer your question (in the original addressing scheme),

- my router interface `10.10.0.1/24` connects to the proxmox interface `10.10.0.10/24`
- my router interface `10.20.0.1/24` connects to the proxmox interface `10.20.0.10/24`
- there are other networks on the `vmbr1.x` vlans.
- on the routers, there are matching vlans with tagged/trunked connection.
- for the `10.10.0.0/16` network, `10.10.30.0/24` is restricted to vlan30 and so forth
- the `10.10.0.0/24` network then uses vlan1000 as its main vlan.

I'm making some basic mistakes here with the addressing/subnetting. I'm trying to subdivide a large /16 network into subnets, but I think the connection between the DD-WRT router and the Proxmox `vmbr1` needs to be on a separate network. But i can't quite figure out how to get this to work without setting the `vmbr1` gateway to be outside the subnets in its VLANs.

e.g. DD-WRT `10.9.0.1/24` would talk to a `10.9.0.10/24` on a proxmox ethernet interface/bridge which would then send layer3 traffic to another network which contains the VLANs on vmbr1.

I'm trying to get route summarization on my DD-WRT routers. I looked into OSPF, but i just don't have enough routers to justify it -- the OSPF "autonomous system" would consist of only the backbone area. This changes if I can get VyOS running inside Proxmox. If I can't, then OSPF must be provided with information about the networks inside Proxmox (via LSA's 5 and 7). This means I need to manually configure each network/subnet I add from within Proxmox into the DD-WRT router adjacent to the Proxmox `10.10.0.0 / 16` or `10.20.0.0 / 16` network. It's very likely that I'm overlooking some fairly obvious possibilities here.

My main goal is to create separation between groups of VM's by using networking. I may be overthinking it, since I could do most of this via Proxmox Node & VM firewalls. The downside of this approach is that I have to manually write `iptables` forwarding rules on the DD-WRT devices when VM addresses, protocols or traffic flows change.

 

[cxor]

The main problem I'm having when I use SDN Zones/VNets is that they don't seem to be set up within the VRF contexts.

SDN Zones

vlan: zvrftest
        bridge vmbr7
        ipam pve

vlan: zvrfdesk
        bridge vmbr1
        ipam pve

SDN VNets. i have tried both vlanaware and not.

vnet: vnet4010
        zone zvrftest
        tag 4010
        vlanaware 1

vnet: vnet4044
        zone zvrftest
        tag 4044
        vlanaware 1

vnet: vnet4064
        zone zvrftest
        tag 4064

vnet: vdesk10
        zone zvrfdesk
        tag 10

vnet: vdesk44
        zone zvrfdesk
        tag 44

vnet: vdesk64
        zone zvrfdesk
        tag 64

And the current interfaces I have configured are below. I needed to change the addresses again to more closely reflect the configuration I have. (sorry to be confusing)

For the VLANS in vmbr7, I have tried the SDN approach and the VLAN approach. Once I have the routing working, I should be able to write a few simple Ansible playbooks to ping & traceroute specific addresses from various VM's. But for now, this isn't really an option.

I am also unsure of where, specifically, the vrf directives are needed.

auto lo
iface lo inet loopback

iface enp1s0 inet manual

iface enp2s0 inet manual

iface enp3s0 inet manual

iface enp4s0 inet manual

auto vrftest
iface vrftest
        vrf-table auto

auto vmbr7
iface vmbr7 inet static
        #address 10.230.16.10/24
        gateway 10.230.16.1
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        vrf vrftest

auto vmbr7.22
iface vmbr7.22 inet static
        address 10.230.16.10/24

auto vmbr0
iface vmbr0 inet static
        address 10.210.0.10/24
        gateway 10.210.0.1
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094


   
auto vrf1
iface vrf1
        vrf-table auto

auto vmbr1
iface vmbr1 inet static
        address 10.50.0.10/24
        gateway 10.50.0.1
        bridge-ports enp4s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        vrf vrf1

# auto vmbr1.44
# iface vmbr1.44 inet static
#       address 10.50.44.10/24
#       vrf vrf1

# auto vmbr1.64
# iface vmbr1.50 inet static
#       address 10.50.64.10/24
#       vrf vrf1

# auto vmbr1.10
# iface vmbr1.10 inet static
#       address 10.50.10.10/24
#       vrf vrf1

And here are the SDN-confgured interfaces:

auto vdesk10
iface vdesk10
        bridge_ports vmbr1.10
        bridge_stp off
        bridge_fd 0

auto vdesk44
iface vdesk44
        bridge_ports vmbr1.44
        bridge_stp off
        bridge_fd 0

auto vdesk64
iface vdesk64
        bridge_ports vmbr1.64
        bridge_stp off
        bridge_fd 0

auto vnet4010
iface vnet4010
        bridge_ports vmbr7.4010
        bridge_stp off
        bridge_fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vnet4044
iface vnet4044
        bridge_ports vmbr7.4044
        bridge_stp off
        bridge_fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vnet4064
iface vnet4064
        bridge_ports vmbr7.4064
        bridge_stp off
        bridge_fd 0

I am getting a router soon. It will be a layer 2 or 3 device.

 

[spirit]

sdn currently don't have vrf implemented (only in evpn zone plugin).

just a question, why do you need to use routing on proxmox, if you transport vlans to your routers ?
Can't you simply pout on your router differents vlans with gateway for your vms ?

 

[jnoxon]

Since it's been about a year, I figured I'd ask if SDN works with VRF now. I'd like to use SDN, but it's a hard requirement that my SDN uses a different gateway.

 

[spirit]

Since it's been about a year, I figured I'd ask if SDN works with VRF now. I'd like to use SDN, but it's a hard requirement that my SDN uses a different gateway.

currently, only evpn is supported vrf, but I'm planning to add it for simple zone too in coming weeks.