Advice on SDN, with PVE cluster nodes on different networks

L

lmasarati

New Member
Apr 17, 2024
1
0
1
Italy
www.outsys.com
Hello everybody,

I'm currently planning the deployment of a PVE cluster, span over multiple datacenters so I'm testing a SDN configuration that allows to define a dedicated subnet for the VMs, conceptually like a virtual datacenter for VMs, with full L3 routing with the real DCs.


1729935517494.png

  • I'm using PVE 8.2.7
  • Each datacenter is already connected with the others through IPSec tunnels and static routing.
  • Each datacenter has a node of the PVE cluster
  • The VMs hosted by each node of the PVE cluster:
    • must be connected to the same dedicated SDN
    • must be able to reach the VMs hosted by the other PVE nodes of the same cluster
    • must be able to reach the real servers in both datacenters (without using SNAT)
    • must be reachable by real servers from both datacenters
Following a tutorial and the official documentation I first configured the SDN following the VXLAN model, as a result:
  • the communication between VMs hosted by different PVE nodes is OK, as expected
  • I can see traffic flowing in UDP/4789 tunnel between PVE nodes, as expected
  • the communication from VMs to servers in real DCs is not working, as expected
  • the communication from servers in real DCs to VMs is not working, as expected
  • the VXLAN network is shared between remote PVE cluster nodes, but isolated from networks in datacenters
  • ICMP is NOT working between VMs and their GW (192.168.64.1) (is it expected ?)
Then I re-configured the SDN following the EVPN model.
Each datacenter has static routes to route the 192.168.64.0/24 network into the nearest node of the PVE cluster, and now:
  • the communication between VMs hosted by different PVE nodes is NOT working
  • I CANNOT see traffic flowing in UDP/4789 tunnel between PVE nodes
  • the communication from VMs to servers in real DCs is working only on same site
    • from VMs hosted by PVE01 to DC1
    • from VMs hosted by PVE03 to DC3
  • the communication from servers in real DCs to VMs is working only on same site
    • from DC1 to VMs hosted by PVE01
    • from DC3 to VMs hosted by PVE03
  • ICMP is working between VMs and their GW (192.168.64.1)
This is the current configuration of SDN in cluster
Code: 
root@pve01:~# cat /etc/pve/sdn/*
evpn: evpnctrl
        asn 65000
        peers 192.168.127.78,192.168.230.12

subnet: evpnzone-192.168.64.0-24
        vnet evpnnet
        gateway 192.168.64.1

vnet: evpnnet
        zone evpnzone
        alias EVPN Network
        tag 11000

evpn: evpnzone
        controller evpnctrl
        vrf-vxlan 10000
        exitnodes pve03,pve01
        ipam pve
        mac BC:24:11:F9:5F:A3

root@pve01:~#

This is the network configuration on PVE01 node for EVPN model:
Code: 
root@pve01:~# cat /etc/network/interfaces.d/sdn
#version:47

auto evpnnet
iface evpnnet
        address 192.168.64.1/24
        hwaddress BC:24:11:F9:5F:A3
        bridge_ports vxlan_evpnnet
        bridge_stp off
        bridge_fd 0
        mtu 1450
        alias EVPN Network
        ip-forward on
        arp-accept on
        vrf vrf_evpnzone

auto vrf_evpnzone
iface vrf_evpnzone
        vrf-table auto
        post-up ip route del vrf vrf_evpnzone unreachable default metric 4278198272

auto vrfbr_evpnzone
iface vrfbr_evpnzone
        bridge-ports vrfvx_evpnzone
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_evpnzone

auto vrfvx_evpnzone
iface vrfvx_evpnzone
        vxlan-id 10000
        vxlan-local-tunnelip 192.168.230.12
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_evpnnet
iface vxlan_evpnnet
        vxlan-id 11000
        vxlan-local-tunnelip 192.168.230.12
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450
root@pve01:~#

root@pve01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether e4:43:4b:61:f0:b0 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0np0
3: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:61:f0:b1 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1np1
4: enp132s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f4:c7:aa:43:1b:08 brd ff:ff:ff:ff:ff:ff
5: eno3np2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:61:f0:b2 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f2np2
6: enp132s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f4:c7:aa:43:1b:09 brd ff:ff:ff:ff:ff:ff
7: eno4np3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:61:f0:b3 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f3np3
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:43:4b:61:f0:b0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.230.12/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::e643:4bff:fe61:f0b0/64 scope link
       valid_lft forever preferred_lft forever
10: evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
    link/ether bc:24:11:f9:5f:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.1/24 scope global evpnnet
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fef9:5fa3/64 scope link
       valid_lft forever preferred_lft forever
11: vrf_evpnzone: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:c2:31:9d:7b:25 brd ff:ff:ff:ff:ff:ff
13: vrfbr_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
    link/ether 9a:5c:20:03:ed:b6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a04c:afff:fec0:4e12/64 scope link
       valid_lft forever preferred_lft forever
14: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master evpnnet state UNKNOWN group default qlen 1000
    link/ether 2a:af:f4:50:3e:2b brd ff:ff:ff:ff:ff:ff
25: vxlan_evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master evpnnet state UNKNOWN group default qlen 1000
    link/ether 16:6e:62:87:b7:4f brd ff:ff:ff:ff:ff:ff
26: vrfvx_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrfbr_evpnzone state UNKNOWN group default qlen 1000
    link/ether 9a:5c:20:03:ed:b6 brd ff:ff:ff:ff:ff:ff
root@pve01:~#

root@pve01:~# ip r
default via 192.168.230.1 dev vmbr0 proto kernel onlink
192.168.64.0/24 nhid 47 dev evpnnet proto bgp metric 20
192.168.230.0/24 dev vmbr0 proto kernel scope link src 192.168.230.12
root@pve01:~#

This is the current configuration on PVE03 node:
Code: 
root@pve03:~# cat /etc/network/interfaces.d/sdn
#version:47

auto evpnnet
iface evpnnet
        address 192.168.64.1/24
        hwaddress BC:24:11:F9:5F:A3
        bridge_ports vxlan_evpnnet
        bridge_stp off
        bridge_fd 0
        mtu 1450
        alias EVPN Network
        ip-forward on
        arp-accept on
        vrf vrf_evpnzone

auto vrf_evpnzone
iface vrf_evpnzone
        vrf-table auto
        post-up ip route del vrf vrf_evpnzone unreachable default metric 4278198272

auto vrfbr_evpnzone
iface vrfbr_evpnzone
        bridge-ports vrfvx_evpnzone
        bridge_stp off
        bridge_fd 0
        mtu 1450
        vrf vrf_evpnzone

auto vrfvx_evpnzone
iface vrfvx_evpnzone
        vxlan-id 10000
        vxlan-local-tunnelip 192.168.127.78
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450

auto vxlan_evpnnet
iface vxlan_evpnnet
        vxlan-id 11000
        vxlan-local-tunnelip 192.168.127.78
        bridge-learning off
        bridge-arp-nd-suppress on
        mtu 1450
root@pve03:~#

root@pve03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 7c:83:34:be:1c:ab brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:83:34:be:1c:ac brd ff:ff:ff:ff:ff:ff
4: wlo1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b0:dc:ef:b8:14:e7 brd ff:ff:ff:ff:ff:ff
    altname wlp0s20f3
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7c:83:34:be:1c:ab brd ff:ff:ff:ff:ff:ff
    inet 192.168.127.78/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::7e83:34ff:febe:1cab/64 scope link
       valid_lft forever preferred_lft forever
7: evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
    link/ether bc:24:11:f9:5f:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.1/24 scope global evpnnet
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fef9:5fa3/64 scope link
       valid_lft forever preferred_lft forever
8: vrf_evpnzone: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:81:16:88:8e:36 brd ff:ff:ff:ff:ff:ff
10: vrfbr_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_evpnzone state UP group default qlen 1000
    link/ether f2:64:0b:58:dc:e4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4cc:acff:fee5:52db/64 scope link
       valid_lft forever preferred_lft forever
11: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master evpnnet state UNKNOWN group default qlen 1000
    link/ether c6:27:b2:20:8d:00 brd ff:ff:ff:ff:ff:ff
20: vxlan_evpnnet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master evpnnet state UNKNOWN group default qlen 1000
    link/ether 8a:a2:62:d2:b8:3d brd ff:ff:ff:ff:ff:ff
21: vrfvx_evpnzone: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrfbr_evpnzone state UNKNOWN group default qlen 1000
    link/ether f2:64:0b:58:dc:e4 brd ff:ff:ff:ff:ff:ff
root@pve03:~#

root@pve03:~# ip r
default via 192.168.127.1 dev vmbr0 proto kernel onlink
192.168.64.0/24 nhid 31 dev evpnnet proto bgp metric 20
192.168.127.0/24 dev vmbr0 proto kernel scope link src 192.168.127.78
root@pve03:~#

IMHO the strangest part is that in EVPN model I never see any traffic flowing in UDP/4789 tunnel between PVE nodes...
I thought that EVPN mode should be based on the same UDP/4789 tunnel as VXLAN mode for intra-cluster communications, isn't it?
Am I missing something?

Thanks in advance.
 
Last edited:
You must log in or register to reply here.
Top Bottom