proxmox 7.0 sdn beta test

[Ben B]

G'day folks,

For reasons a plenty, looking at making some additions including:

- Controller: different ASN between host and peer

root@pmx03:~# cat /etc/pve/sdn/*
evpn: ctrlr1
    asn 65004
    peers 10.61.2.142
    gateway-nodes pmx03
root@pmx03:~#

pmx03# sh run
Building configuration...

Current configuration:
!
frr version 7.2.1
frr defaults traditional
hostname pmx03
log syslog informational
service integrated-vtysh-config
!
router bgp 65004
 bgp router-id 10.24.1.111
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor 10.61.2.142 remote-as 65004
 !
 address-family l2vpn evpn
  neighbor 10.61.2.142 activate
  advertise-all-vni
 exit-address-family
!
line vty
!
end
pmx03#

- Controller: capability for filters to not send default, for options to not have the below

142         @controller_config = ();
 143         #add default originate to announce 0.0.0.0/0 type5 route in evpn
 144         push @controller_config, "default-originate ipv4";
 145         push @controller_config, "default-originate ipv6";
 146         push(@{$config->{frr}->{router}->{"bgp $asn vrf $vrf"}->{"address-family"}->{"l2vpn evpn"}}, @controller_config);

Is/has anyone been working on this already but it just hasn't yet been pushed to the non-enterprise repositories?

Cheers,
Ben.

 

[alyarb]

VLAN mode is working well in terms of creating/applying the config.

So I create a VXLAN zone called "SDN" with MTU 8950 and all my hosts' vmbr0 addresses in the peer list.

Then I create a net called testNet, tag 9999 and the rest on auto. When I hit apply, the "pending" zone turns to error.

Where can I see the error message?

Hotplug changing the NIC's bridge does not work. Tested on Windows and pfSense VM. Qemu-ga is working, but should not be required.

Hotplug error:

Parameter verification failed. (400)

net3: hotplug problem - can't add ovs port 'tap9902i3' - command '/usr/bin/ovs-vsctl -- add-port testNet tap9902i3' failed: open3: exec of /usr/bin/ovs-vsctl -- add-port testNet tap9902i3 failed: No such file or directory at /usr/share/perl5/PVE/Tools.pm line 444.

Just to add to this, if I attach my vNet to a NIC on a powered off machine, and then start it, the error is different:

bridge 'testNet' does not exist
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
TASK ERROR: start failed: QEMU exited with code 1

What should I be looking for? The Zone is showing as available on all hosts.

Should "testNet" be showing on the Network pane of each host?

Thanks for your help.

 

[David Hooton]

G'day folks,

For reasons a plenty, looking at making some additions including:

- Controller: different ASN between host and peer

root@pmx03:~# cat /etc/pve/sdn/*
evpn: ctrlr1
    asn 65004
    peers 10.61.2.142
    gateway-nodes pmx03
root@pmx03:~#

pmx03# sh run
Building configuration...

Current configuration:
!
frr version 7.2.1
frr defaults traditional
hostname pmx03
log syslog informational
service integrated-vtysh-config
!
router bgp 65004
bgp router-id 10.24.1.111
no bgp default ipv4-unicast
coalesce-time 1000
neighbor 10.61.2.142 remote-as 65004
!
address-family l2vpn evpn
  neighbor 10.61.2.142 activate
  advertise-all-vni
exit-address-family
!
line vty
!
end
pmx03#

- Controller: capability for filters to not send default, for options to not have the below

142         @controller_config = ();
143         #add default originate to announce 0.0.0.0/0 type5 route in evpn
144         push @controller_config, "default-originate ipv4";
145         push @controller_config, "default-originate ipv6";
146         push(@{$config->{frr}->{router}->{"bgp $asn vrf $vrf"}->{"address-family"}->{"l2vpn evpn"}}, @controller_config);

Is/has anyone been working on this already but it just hasn't yet been pushed to the non-enterprise repositories?

Cheers,
Ben.

+1 this is absolutely a requirement for our EVPN implementation as well.

 

[spirit]

VLAN mode is working well in terms of creating/applying the config.

So I create a VXLAN zone called "SDN" with MTU 8950 and all my hosts' vmbr0 addresses in the peer list.

Then I create a net called testNet, tag 9999 and the rest on auto. When I hit apply, the "pending" zone turns to error.

Where can I see the error message?

Hotplug changing the NIC's bridge does not work. Tested on Windows and pfSense VM. Qemu-ga is working, but should not be required.

Hotplug error:

Parameter verification failed. (400)

net3: hotplug problem - can't add ovs port 'tap9902i3' - command '/usr/bin/ovs-vsctl -- add-port testNet tap9902i3' failed: open3: exec of /usr/bin/ovs-vsctl -- add-port testNet tap9902i3 failed: No such file or directory at /usr/share/perl5/PVE/Tools.pm line 444.

Hi,
you need to add "source /etc/network/interfaces.d/*" in /etc/network/interfaces.

(I really need to find a way to auto add it, because you're not the first to forget it)

The ovs_port hotplug bug, is because I think It don't found the bridge, and try to fallback to ovs. (I have already fixed that in a new coming version, where I don't display the vnet is the config is not correctly loaded)

 

[alyarb]

Thanks spirit. I knew I read that section in the doc and it just didn't click, but yes it was easy to forget.

In the meantime I know it's not pretty but it's the least amount of work

sed -i '/interfaces.d/d' /etc/network/interfaces; printf 'source /etc/network/interfaces.d/*' >> /etc/network/interfaces

VXLAN zones working good, thanks a ton

 

[spirit]

@Ben B

Hi ben,

- Controller: different ASN between host and peer

Currently I'm only manage ibgp (same AS), but I'm planning to add e-bgp soon where you'll be able to define different ASN.
Also, I think I'll add the option to define for each host, a different peer+ASN.
(for a full l3 network, where different proxmox hosts peer directly with their attached router)

Should it work for you need ?

- "Controller: capability for filters to not send default, for options to not have the below",

currently is enable only when a node is defined as a outside gateway.
Do you want to be able to specify a custom subnet list instead a default ?

@Ben B @David Hooton

Could you describe your current needs and setup ?
I don't have yet too much testers for evpn, so I'll happy to implement missing features.


Edit : Forget to add it in documentation, but it's possible to override frr.conf for each host, with creating an /etc/frr/frr.conf.local
If you have complex config need, it can be usefull.
(But I would like to be able to do main user need through the gui)

 

[cyruspy]

mmm, ok, I think I understand.
Why not simply do , instead your conf:

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp5s4f0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.100
iface vmbr0.100 inet manual
    vlan-id 100


auto vmbr1
iface vmbr1 inet static
    address 10.2.0.223/24
    gateway 10.2.0.1
    bridge-ports enp6s5f0 enp6s5f1 vmbr0.100
    bridge-stp off
    bridge-fd 0
#Downlink para gest

this (tagging vlan on physical interfaces)

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp5s4f0  enp6s5f0.100 enp6s5f1.100
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.100
iface vmbr0.100 inet static
    address 10.2.0.223/24
    gateway 10.2.0.1

or using vlan-aware feature (with ifupdown2 package)

auto enp6s5f0
iface enp6s5f0
    bridge-access 100

auto enp6s5f1
iface enp6s5f1
    bridge-access 100

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp5s4f0  enp6s5f0 enp6s5f1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.100
iface vmbr0.100 inet static
    address 10.2.0.223/24
    gateway 10.2.0.1

Thanks for the suggestion, will check in depth once I get home. Could the current config the source of the duplicated packages?.

Will be out some weeks and don't want to kill my remote access if the changes fail somehow

 

[spirit]

Thanks for the suggestion, will check in depth once I get home. Could the current config the source of the duplicated packages?.

Will be out some weeks and don't want to kill my remote access if the changes fail somehow

I really can tell, but the tagged bridge inside another bridge is not too well tested, so I really can't tell the behaviour.
Duplicated packets is some kind of network loop, when sent packets are coming back.

 

[David Hooton]

@Ben B @David Hooton

Could you describe your current needs and setup ?
I don't have yet too much testers for evpn, so I'll happy to implement missing features.


Edit : Forget to add it in documentation, but it's possible to override frr.conf for each host, with creating an /etc/frr/frr.conf.local
If you have complex config need, it can be usefull.
(But I would like to be able to do main user need through the gui)

Hey!

This is what our initial setup is going to look like. Every device in our network has a unique ASN.

Once we've tested fully, every host will be multihomed to at least 2 switches.


In an ideal world we would be able to do a type 5 EVPN interface for the management interface. Being able to define prefix filter lists on type 5 routing tables is a definite must have as we don't want hypervisor chassis announcing themselves as default gateways.

 

[spirit]

Hey!

This is what our initial setup is going to look like. Every device in our network has a unique ASN.

Once we've tested fully, every host will be multihomed to at least 2 switches.

View attachment 20578

ok thanks. I'm going to do same kind of setup at work (but with 2 routers in same AS for 1 proxmox host, with ecmp).
I Think I'll add some kind of controller config in each node network gui., overriding global values of datacenter level.

In an ideal world we would be able to do a type 5 EVPN interface for the management interface. Being able to define prefix filter lists on type 5 routing tables is a definite must have as we don't want hypervisor chassis announcing themselves as default gateways.

evpn network (each zone), are in a separated vrf. So hypervisor themself don't annouce the as default, "until" you define seem as exit gateway for the evpn network. (with a vrf leak between the zone vrf && default vrf). Personnaly, I'm using arista routers with evpn-bgp for the exit.
But I could add something like "announce default, but not theses subnets... Or annouce this subnets only ..." if needed.
Do you have some kind of example of filtering with frr.conf ?

 

[David Hooton]

ok thanks. I'm going to do same kind of setup at work (but with 2 routers in same AS for 1 proxmox host, with ecmp).
I Think I'll add some kind of controller config in each node network gui., overriding global values of datacenter level.

This would be great.

evpn network (each zone), are in a separated vrf. So hypervisor themself anounce announcing themself as default gateway, "until" you define seem as exit gateway for the evpn network. (Personnaly, I'm using arista routers with evpn-bgp for the exit).

Cool - we are using Arista as well.

Do you have some kind of example of filtering with frr.conf ?

I don't at this stage, but it would just be a prefix list with the local interfaces I want to announce listed in it.

 

[spirit]

This would be great.

Cool - we are using Arista as well.

I don't at this stage, but it would just be a prefix list with the local interfaces I want to announce listed in it.

ok, so instead simply announcing a default, you want to just announce a list of prefixes for the type5?

(Note that if you have arista hardware, you can do it directly from them, instead using some proxmox host as exit nodes)

 

[David Hooton]

ok, so instead simply announcing a default, you want to just announce a list of prefixes for the type5?

(Note that if you have arista hardware, you can do it directly from them, instead using some proxmox host as exit nodes)

Yes I would normally only announce local prefixes from the host which would more likely receive default from the Arista's.

We would definitely also filter on the arista's.

 

[spirit]

Yes I would normally only announce local prefixes from the host which would more likely receive default from the Arista's.

I mean, if your arista can do bgp-evpn, it could announce directly the type5 in the evpn, so it'll not leave the overlay, and you don't need to announce prefixes from the host.

 

[fhloston]

I have some thoughts regarding the SDN VXLAN implementation.

The VXLAN interface is configured with vxlan_remoteip <peerips>. To my understanding this means that BUM traffic is replicated from one node to all others. This has implications for scaling.

Another approach would be to use a Multicast group for BUM traffic. In this case one would also not have to specify all peers upon VXLAN creation.

As far as I can tell, ifupdown2 already has support for this: vxlan-mcastgrp 172.16.22.127

Can this be implemented in the gui?

 

[spirit]

I have some thoughts regarding the SDN VXLAN implementation.

The VXLAN interface is configured with vxlan_remoteip <peerips>. To my understanding this means that BUM traffic is replicated from one node to all others. This has implications for scaling.

Another approach would be to use a Multicast group for BUM traffic. In this case one would also not have to specify all peers upon VXLAN creation.

As far as I can tell, ifupdown2 already has support for this: vxlan-mcastgrp 172.16.22.127

Can this be implemented in the gui?

yes, I could add multicast support too if needed. I need to find a way to be able to define the physical interface used, because It can be different on each host.

I also need to find a way to auto compute the multicast address, because you need 1 multicast address for each vxlan-id.


also, if you really need to scale, you could also try to use the bgp-evpn plugin, where you don't have any bum traffic at all. the mac/ip address location are exchanged in the control plane through the bgp protocol, and arp is suppressed.

 

[alyarb]

Hi Spirit... Can we try to enable some of the "illegal" characters on the vNet ID? Why do you think underscore, period, or hyphen should not be allowed? What about increasing the max length to 16?

Thanks

 

[spirit]

Hi Spirit... Can we try to enable some of the "illegal" characters on the vNet ID? Why do you think underscore, period, or hyphen should not be allowed? What about increasing the max length to 16?

Thanks

I need to check about special characters, to see what is autorized in interface name by kernel. (for example @ is already used, dot is for vlan,... ). I'm also already using underscore as split character for some internal things.

About 16 characters, no, I'm currently limiting it to 8 characters, because I need more space for some zone plugin for intermediate names.

That's why you have the alias field, you can set what you want here.

 

[fhloston]

I also need to find a way to auto compute the multicast address, because you need 1 multicast address for each vxlan-id.

actually, to my understanding you don't. you can run all vxlan-ids in the same multicast group. it is counterproductive to spawn multicast groups per vxlan.

Default for linux is max. 20 igmp memberships...
net.ipv4.igmp_max_memberships = 20

also, if you really need to scale, you could also try to use the bgp-evpn plugin, where you don't have any bum traffic at all. the mac/ip address location are exchanged in the control plane through the bgp protocol, and arp is suppressed.

yes, i had a look at bgp-evpn.

currently i am trying to keep complexity as simple as possible and giving every customer his own vlan space inside one customer vxlan.

in the evpn usecase we then have one vxlan for the control plane and another for the data-plane - correct?

 

[spirit]

in the evpn usecase we then have one vxlan for the control plane and another for the data-plane - correct?

Well, control plane have nothing to do with vxlan. That's just mean than indead using arp, brodcast, to fill the mac address table of the bridges, This is done by the controller (here, frr with bgp-evpn protocol).

you can still have multiple vxlan.

but also, you can do vxlan routing (optionnaly,if you define a gateway address on vmbr). And this is done through a vrf, with a specific vxlan. (by zone).

But you can use bgp-evpn for simple vxlan tunnel without any routing.

If you have a lot of hosts, this could make sense (and you don't have to manage a multicast network, which can be a pain with igmp snooping and others multicast stuff)