proxmox 7.0 sdn beta test

[spirit]

Question:

why do you define the bgp controller with "ebgp:1" , with your route-reflector as peer,
as you use the same 65001 asn on both node && route reflector ?

ebgp:1 add " neighbor BGP remote-as external", so the peer should have a different asn.


If you need to do a simple evpn route reflector with ibgp (same asn on each proxmox node), same as, you don't even need to define a bgp controller in the sdn.
Simply create the evpn controller, add add the route reflector as peer.



The bgp controller is mainly used for 2 things:

- If you want to use ebgp (different asn for each proxmox node) for the evpn.

- if you want add an external bgp peer to a specific node. (on an exit-node for example, to forward evpn routes to a classic bgp router)

 

[bitposs]

The vms ipv4 /32 or ipv6 /128 should be announced as evpn type3 routes.
It's possible to advertise the subnets defined in subnets.cfg with evpn type5 routes. It's not available in gui yet (I had send patchs some months ago, but it's not applied, I need to check that). But it should be possible to add the option in the zones.cfg.

OK perfect! For what its worth, here are my package versions (haven't yet upgraded to 7.X as I was having issues with my Intel 10Gig NICs and still have to sort that):

proxmox-ve: 6.4-1 (running kernel: 5.4.162-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-12
pve-kernel-helper: 6.4-12
pve-kernel-5.4.162-1-pve: 5.4.162-2
pve-kernel-5.4.157-1-pve: 5.4.157-1
pve-kernel-5.4.151-1-pve: 5.4.151-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve4~bpo10
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-network-perl: 0.6.0
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.13-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.7-pve1

In your route reflector, you should be able to redistribute evpn routes into your classic bgp routes

Maybe it could help, I have wrote note about frr route reflector last year(s)
https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD
(see Route Reflectors section at the end)

I think I had take info from here:
https://vincent.bernat.ch/en/blog/2017-vxlan-bgp-evpn#using-frr

I don't have tried it since this time, it's almost the same conf that your config, but they are some more option.

All of this is fantastic info, thanks for sharing! I had a good idea it was you that wrote the above documentation (on git), we were using this prior to the SDN, all REALLY useful . The SDN helps streamline this and makes deployments super easy (we're a service provider so isolating subnets per VM is essential and now can be automated ).

Question:

why do you define the bgp controller with "ebgp:1" , with your route-reflector as peer,
as you use the same 65001 asn on both node && route reflector ?

ebgp:1 add " neighbor BGP remote-as external", so the peer should have a different asn.


If you need to do a simple evpn route reflector with ibgp (same asn on each proxmox node), same as, you don't even need to define a bgp controller in the sdn.
Simply create the evpn controller, add add the route reflector as peer.



The bgp controller is mainly used for 2 things:

- If you want to use ebgp (different asn for each proxmox node) for the evpn.

- if you want add an external bgp peer to a specific node. (on an exit-node for example, to forward evpn routes to a classic bgp router)

Yup, my bad. I was in a rush yesterday to get into a meeting and fired off that reply before I reviewed it; but was over complicating everything.

Basically, all I needed was an external BGP peer to forward the evpn routes to my upstream BGP router(s) as they don't support ECMP static routes hence we want to eliminate the single point of failure on the exit node. I know in your above docs you had mentioned using VRRP, but wasn't sure how that'd integrate with the SDN plugin and wanted to keep everything in one place.

This config achieved what we were looking to do:

root@myhostname-24:~# cat /etc/pve/sdn/*.cfg
evpn: evpn400
        asn 65001
        peers 192.168.1.24,192.168.1.25

bgp: bgpmyhostname-24
        asn 65001
        node myhostname-24
        peers 192.168.1.18
        ebgp 1

bgp: bgpmyhostname-25
        asn 65001
        node myhostname-25
        peers 192.168.1.18
        ebgp 1

subnet: evpn400-10.10.10.160-27
        vnet pubnet2
        gateway 10.10.10.190

vnet: pubnet2
        zone evpn400
        tag 14000

evpn: evpn400
        controller evpn400
        vrf-vxlan 10001
        exitnodes myhostname-25,myhostname-24
        ipam pve
        mac 9A:E1:F8:7C:C2:70
        mtu 1450

The evpn subnet is now perfectly passing to 192.168.1.18 (this is test environment, adding a redundant peer on live)!

I cannot say enough good things about your work here @spirit - major kudos! Appreciate the support!

 

[spirit]

Ok great !

Note that I don't work anymore the proxmox6 version of the plugin.
proxmox7 have new frr8 version with some evpn fixes, and sdn evpn plugin have also some new features (like advertise-subnets option).

I need to look at vrrp soon, I don't known yet if it'll manage it with frr directly ou keepalived. I need to look at this. (Some users have requested it when they have non-bgp upstream routers with static routes)

 

[bitposs]

Ok great !

Note that I don't work anymore the proxmox6 version of the plugin.
proxmox7 have new frr8 version with some evpn fixes, and sdn evpn plugin have also some new features (like advertise-subnets option).

I need to look at vrrp soon, I don't known yet if it'll manage it with frr directly ou keepalived. I need to look at this. (Some users have requested it when they have non-bgp upstream routers with static routes)

Thanks for letting me know! I think I'll start install ProxMox 7 on my new clusters going forward and also look at upgrading some of my existing.

Along that note, I played around with using VRRP with FRR but ran into some troubles with my maclan devices, which I created on top of my linux bridge that uses a linux bond in active-backup mode. The multicast packets would only pass with the same virtual router ID, but kept creating loops, regardless my stp settings (same switch VLAN). Changing them to use unique virtual router IDs solved the loop issue, but both stayed in Master mode. I tracked some of this down to my link-local IPV6 links, but eventually decided to just use keep alive.

For those interested in VRRP with KeepAlive:

Perform these steps on Proxmox exit nodes, making sure to modify keepalived.conf accordingly (i.e. master/backup):

Install:

$ sudo apt-get install keepalived

Allow non-local IP binding:

$ sudo echo "net.ipv4.ip_nonlocal_bind=1" > /etc/sysctl.conf

Restart sysctl:

$ sudo sysctl -p

Configure:

$ sudo nano /etc/keepalived/keepalived.conf

# Create VRRP instance
vrrp_instance VRRP_ALPHA {

    # The interface keepalived will manage
    interface vmbr0

    # The initial state to transition to. This option isn't
    # really all that valuable, since an election will occur
    # and the host with the highest priority will become
    # the master. The priority is controlled with the priority
    # configuration directive. MASTER or BACKUP
    state MASTER

    # The virtual router id number to assign the routers to
    virtual_router_id 1

    # The priority to assign to this device. This controls
    # who will become the MASTER and BACKUP for a given
    # VRRP instance. Higher has more weight 1-255.
    priority 255

    # How many seconds to wait until a gratuitous arp is sent
    garp_master_delay 2

    # How often to send out VRRP advertisements
    advert_int 1

    # IP Address of this device
    unicast_src_ip 192.168.1.1

    # IP Address of the peer device
    unicast_peer{
        192.168.1.2
    }

    # Authenticate the remote endpoints via a simple
    # username/password combination
    authentication {
        auth_type AH
        auth_pass monkey
    }
    # The virtual IP addresses to float between nodes. The
    # label statement can be used to bring an interface
    # online to represent the virtual IP.
        virtual_ipaddress {
        192.168.1.3 dev vmbr0 label vmbr0:vip
    }
}

Start service:

$ sudo service keepalived start

Check status for errors:

$ sudo systemctl status keepalived

 

[timiion]

Hi, I am not sure this problem is suitable to ask here. I have a problem with VM to DNS query outside the node. It seems that DNS reply message was tampered under EVPN zone.

Topo: (172.20.132.1)VM --- Node --- DNS Server(10.88.0.1)

The VM network is attached on EVPN zone, there is an SNAT rule to outside node.

iptables SNAT rule

Chain POSTROUTING (policy ACCEPT 351 packets, 33491 bytes)
 pkts bytes target     prot opt in     out     source               destination
 9005  654K MASQUERADE  all  --  *      eno1    172.20.128.0/21      0.0.0.0/0

The DNS reply message was tampered, DNS server port was redirected from 53 to other(here example is 475).

capture with tcpdump on VM

21:32:35.382963 IP 172.20.132.1.39787 > 10.88.0.1.53: 14240+ A? ntp.ubuntu.com. (32)
21:32:35.579912 IP 10.88.0.1.475 > 172.20.132.1.39787: UDP, length 96
21:32:35.579952 IP 172.20.132.1 > 10.88.0.1: ICMP 172.20.132.1 udp port 39787 unreachable, length 132

When I changed the VM network attached to simple zone, DNS reply message was OK.

Thanks for your answer.

 

[spirit]

Hi, I am not sure this problem is suitable to ask here. I have a problem with VM to DNS query outside the node. It seems that DNS reply message was tampered under EVPN zone.

Topo: (172.20.132.1)VM --- Node --- DNS Server(10.88.0.1)

The VM network is attached on EVPN zone, there is an SNAT rule to outside node.

iptables SNAT rule

Chain POSTROUTING (policy ACCEPT 351 packets, 33491 bytes)
 pkts bytes target     prot opt in     out     source               destination
 9005  654K MASQUERADE  all  --  *      eno1    172.20.128.0/21      0.0.0.0/0

The DNS reply message was tampered, DNS server port was redirected from 53 to other(here example is 475).

capture with tcpdump on VM

21:32:35.382963 IP 172.20.132.1.39787 > 10.88.0.1.53: 14240+ A? ntp.ubuntu.com. (32)
21:32:35.579912 IP 10.88.0.1.475 > 172.20.132.1.39787: UDP, length 96
21:32:35.579952 IP 172.20.132.1 > 10.88.0.1: ICMP 172.20.132.1 udp port 39787 unreachable, length 132

When I changed the VM network attached to simple zone, DNS reply message was OK.

Thanks for your answer.

I m currently on holiday , i ll check that next week.

 

[cyruspy]

Hello, is there any tooling to troubleshoot VXLAN zones?. I have the configuration applied in two nodes, but once I move a VM to the second node, connectivity with a VM2 that stays in the original node is lost.

 

[spirit]

Hello, is there any tooling to troubleshoot VXLAN zones?. I have the configuration applied in two nodes, but once I move a VM to the second node, connectivity with a VM2 that stays in the original node is lost.

I don't think they are any tool for debugging kernel vxlan tunnel yet.
the 2 vms are on the same vnet ?
do you use proxmox firewall ? if yes, is the port 4789 openned between the 2 nodes ?

 

[cyruspy]

I don't think they are any tool for debugging kernel vxlan tunnel yet.
the 2 vms are on the same vnet ?
do you use proxmox firewall ? if yes, is the port 4789 openned between the 2 nodes ?

Interestingly enough, I see with tcpdump outgoing traffic on both sides but no incoming traffic. I haven't setup any firewall rules so far. Ping works with "no fragment" and size=1600 bytes (I have MTU=9000 on the NIC and switch side).

edit: added a rule allowing UDP traffic on 4789, but no joy.

 

[cyruspy]

The switches also run VXLAN as TEPs between themselves. Could that be the issue?

 

[spirit]

Ping works with "no fragment" and size=1600 bytes (I have MTU=9000 on the NIC and switch side).

Interestingly enough, I see with tcpdump outgoing traffic on both sides but no incoming traffic.

That"s really strange indeed.

The switches also run VXLAN as TEPs between themselves. Could that be the issue?

I don't known if it's a problem.. (maybe they "intercept" vxlan frame ??) . do you have tried with a simple switch ? (or a cross-cable between 2 servers).

For testing, maybe can you try to enable ipsec tunnel, to hide vxlan to your switches ?
https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#_vxlan_ipsec_encryption

 

[cyruspy]

That"s really strange indeed.



I don't known if it's a problem.. (maybe they "intercept" vxlan frame ??) . do you have tried with a simple switch ? (or a cross-cable between 2 servers).

For testing, maybe can you try to enable ipsec tunnel, to hide vxlan to your switches ?
https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#_vxlan_ipsec_encryption

it's a remote location, I can't change the connectivity. Will look into de IPSEC alternative, the DCN team assures the switches shouldn't mess with the VXLAN traffic on the switches.