Advice on SDN, with PVE cluster nodes on different networks

L

lmasarati

New Member
Apr 17, 2024
1
0
0
Italy
www.outsys.com
Hello everybody,

I'm currently planning the deployment of a PVE cluster, span over multiple datacenters so I'm testing a SDN configuration that allows to define a dedicated subnet for the VMs, conceptually like a virtual datacenter for VMs, with full L3 routing with the real DCs.


1729935517494.png

  • I'm using PVE 8.2.7
  • Each datacenter is already connected with the others through IPSec tunnels and static routing.
  • Each datacenter has a node of the PVE cluster
  • The VMs hosted by each node of the PVE cluster:
    • must be connected to the same dedicated SDN
    • must be able to reach the VMs hosted by the other PVE nodes of the same cluster
    • must be able to reach the real servers in both datacenters (without using SNAT)
    • must be reachable by real servers from both datacenters
Following a tutorial and the official documentation I first configured the SDN following the VXLAN model, as a result:
  • the communication between VMs hosted by different PVE nodes is OK, as expected
  • I can see traffic flowing in UDP/4789 tunnel between PVE nodes, as expected
  • the communication from VMs to servers in real DCs is not working, as expected
  • the communication from servers in real DCs to VMs is not working, as expected
  • the VXLAN network is shared between remote PVE cluster nodes, but isolated from networks in datacenters
  • ICMP is NOT working between VMs and their GW (192.168.64.1) (is it expected ?)
Then I re-configured the SDN following the EVPN model.
Each datacenter has static routes to route the 192.168.64.0/24 network into the nearest node of the PVE cluster, and now:
  • the communication between VMs hosted by different PVE nodes is NOT working
  • I CANNOT see traffic flowing in UDP/4789 tunnel between PVE nodes
  • the communication from VMs to servers in real DCs is working only on same site
    • from VMs hosted by PVE01 to DC1
    • from VMs hosted by PVE03 to DC3
  • the communication from servers in real DCs to VMs is working only on same site
    • from DC1 to VMs hosted by PVE01
    • from DC3 to VMs hosted by PVE03
  • ICMP is working between VMs and their GW (192.168.64.1)
This is the current configuration of SDN in cluster
Code: 
root@pve01:~# cat /etc/pve/sdn/*
evpn: evpnctrl
        asn 65000
        peers 192.168.127.78,192.168.230.12

subnet: evpnzone-192.168.64.0-24
        vnet evpnnet
        gateway 192.168.64.1

vnet: evpnnet
        zone evpnzone
        alias EVPN Network
        tag 11000

evpn: evpnzone
        controller evpnctrl
        vrf-vxlan 10000
        exitnodes pve03,pve01
        ipam pve
        mac BC:24:11:F9:5F:A3

root@pve01:~#

This is the network configuration on PVE01 node for EVPN model:
Code: 
root@pve01:~# cat /etc/network/interfaces.d/sdn
#version:47

auto evpnnet
iface evpnnet
        address 192.168.64.1/24
        hwaddress BC:24:11:F9:5F:A3
        bridge_ports vxlan_evpnnet
        bridge_stp off
        bridge_fd 0
        mtu 1450
        alias EVPN Network
        ip-forward on
        arp-accept on
        vrf vrf_evpnzone

auto vrf_evpnzone
iface vrf_evpnzone
        vrf-table auto
        post-up ip route del vrf vrf_evpnzone unreachable default metric 4278198272

auto vrfbr_evpnzone
iface vrfbr_evpnzone
        bridge-ports vrfvx_evpnzone
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_evpnzone

auto vrfvx_evpnzone
iface vrfvx_evpnzone
        vxlan-id 10000
        vxlan-local-tunnelip 192.168.230.12
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_evpnnet
iface vxlan_evpnnet
        vxlan-id 11000
        vxlan-local-tunnelip 192.168.230.12
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450
root@pve01:~#

root@pve01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether e4:43:4b:61:f0:b0 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0np0
3: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:61:f0:b1 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1np1
4: enp132s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f4:c7:aa:43:1b:08 brd ff:ff:ff:ff:ff:ff
5: eno3np2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:61:f0:b2 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f2np2
6: enp132s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f4:c7:aa:43:1b:09 brd ff:ff:ff:ff:ff:ff
7: eno4np3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:61:f0:b3 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f3np3
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:43:4b:61:f0:b0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.230.12/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::e643:4bff:fe61:f0b0/64 scope link
       valid_lft forever preferred_lft forever
10: evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
    link/ether bc:24:11:f9:5f:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.1/24 scope global evpnnet
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fef9:5fa3/64 scope link
       valid_lft forever preferred_lft forever
11: vrf_evpnzone: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:c2:31:9d:7b:25 brd ff:ff:ff:ff:ff:ff
13: vrfbr_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
    link/ether 9a:5c:20:03:ed:b6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a04c:afff:fec0:4e12/64 scope link
       valid_lft forever preferred_lft forever
14: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master evpnnet state UNKNOWN group default qlen 1000
    link/ether 2a:af:f4:50:3e:2b brd ff:ff:ff:ff:ff:ff
25: vxlan_evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master evpnnet state UNKNOWN group default qlen 1000
    link/ether 16:6e:62:87:b7:4f brd ff:ff:ff:ff:ff:ff
26: vrfvx_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrfbr_evpnzone state UNKNOWN group default qlen 1000
    link/ether 9a:5c:20:03:ed:b6 brd ff:ff:ff:ff:ff:ff
root@pve01:~#

root@pve01:~# ip r
default via 192.168.230.1 dev vmbr0 proto kernel onlink
192.168.64.0/24 nhid 47 dev evpnnet proto bgp metric 20
192.168.230.0/24 dev vmbr0 proto kernel scope link src 192.168.230.12
root@pve01:~#

This is the current configuration on PVE03 node:
Code: 
root@pve03:~# cat /etc/network/interfaces.d/sdn
#version:47

auto evpnnet
iface evpnnet
        address 192.168.64.1/24
        hwaddress BC:24:11:F9:5F:A3
        bridge_ports vxlan_evpnnet
        bridge_stp off
        bridge_fd 0
        mtu 1450
        alias EVPN Network
        ip-forward on
        arp-accept on
        vrf vrf_evpnzone

auto vrf_evpnzone
iface vrf_evpnzone
        vrf-table auto
        post-up ip route del vrf vrf_evpnzone unreachable default metric 4278198272

auto vrfbr_evpnzone
iface vrfbr_evpnzone
        bridge-ports vrfvx_evpnzone
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_evpnzone

auto vrfvx_evpnzone
iface vrfvx_evpnzone
        vxlan-id 10000
        vxlan-local-tunnelip 192.168.127.78
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_evpnnet
iface vxlan_evpnnet
        vxlan-id 11000
        vxlan-local-tunnelip 192.168.127.78
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450
root@pve03:~#

root@pve03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 7c:83:34:be:1c:ab brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:83:34:be:1c:ac brd ff:ff:ff:ff:ff:ff
4: wlo1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b0:dc:ef:b8:14:e7 brd ff:ff:ff:ff:ff:ff
    altname wlp0s20f3
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7c:83:34:be:1c:ab brd ff:ff:ff:ff:ff:ff
    inet 192.168.127.78/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::7e83:34ff:febe:1cab/64 scope link
       valid_lft forever preferred_lft forever
7: evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
    link/ether bc:24:11:f9:5f:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.1/24 scope global evpnnet
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fef9:5fa3/64 scope link
       valid_lft forever preferred_lft forever
8: vrf_evpnzone: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:81:16:88:8e:36 brd ff:ff:ff:ff:ff:ff
10: vrfbr_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
    link/ether f2:64:0b:58:dc:e4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4cc:acff:fee5:52db/64 scope link
       valid_lft forever preferred_lft forever
11: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master evpnnet state UNKNOWN group default qlen 1000
    link/ether c6:27:b2:20:8d:00 brd ff:ff:ff:ff:ff:ff
20: vxlan_evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master evpnnet state UNKNOWN group default qlen 1000
    link/ether 8a:a2:62:d2:b8:3d brd ff:ff:ff:ff:ff:ff
21: vrfvx_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrfbr_evpnzone state UNKNOWN group default qlen 1000
    link/ether f2:64:0b:58:dc:e4 brd ff:ff:ff:ff:ff:ff
root@pve03:~#

root@pve03:~# ip r
default via 192.168.127.1 dev vmbr0 proto kernel onlink
192.168.64.0/24 nhid 31 dev evpnnet proto bgp metric 20
192.168.127.0/24 dev vmbr0 proto kernel scope link src 192.168.127.78
root@pve03:~#

IMHO the strangest part is that in EVPN model I never see any traffic flowing in UDP/4789 tunnel between PVE nodes...
I thought that EVPN mode should be based on the same UDP/4789 tunnel as VXLAN mode for intra-cluster communications, isn't it?
Am I missing something?

Thanks in advance.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back